[VP] Declaration and docs for vp.select intrinsic

[simoll]

Summary

llvm.vp.select extends the regular select instruction with an explicit
vector length (%evl).

All lanes with indexes at and above %evl are
undefined. Lanes below %evl are taken from the first input where the
mask is true and from the second input otherwise.

[simoll]

FYI This is also intended for the vp passthru idiom.
vp.select does not use the %evl like the other intrinsics. Instead, everything that is masked out comes from the second input.

%data = vp.gather(%ptr, %mask, %evl)
%gather_with_passthru = vp.select(%mask, %data, %passthru_value, %evl)

[simoll]

• Added i1 tail_passthru immediate to set passthru behavior above the explicit vector length.

[rogfer01]

For the case of pass-thru, does it make sense to give it a different name like vp.overwrite / vp.insert / vp.update / vp.merge (in lack of better names). When there is pass through, this intrinsic looks to me it can be understood as first taking the whole on_false value and then, to build the result, selectively (as defined by the mask below the EVL) replacing elements with the corresponding values from on_true.

Then the case without pass-thru I'd call it vp.select because this does seem the obvious extension from the IR instruction select to VPred.

I understood from an earlier VPred sync call that you already had considered this scheme with two intrinsics and it had some drawbacks, right? Mind to elaborate? Perhaps a vectorizer will mostly always use vp.merge and never vp.select so the latter is less interesting to have?

llvm/docs/LangRef.rst

17794

I might be reading this wrong, shouldn't it be

The pivot creates a mask, %pivotMask, with all elements ``0 <= i < %pivot`` set to ``1``

(i.e. swap %pivot and %pivotMask occurrences)

17827

I think this should be %m (instead of %M)

[simoll]

In D105351#2919476, @rogfer01 wrote:

For the case of pass-thru, does it make sense to give it a different name like vp.overwrite / vp.insert / vp.update / vp.merge (in lack of better names). When there is pass through, this intrinsic looks to me it can be understood as first taking the whole on_false value and then, to build the result, selectively (as defined by the mask below the EVL) replacing elements with the corresponding values from on_true.

I tend to agree. Mostly, i am worried that the intrinsic id is no longer sufficient to know whether everything above %evl can be ignored and then there is that extra parameter. That's unnecessarily irregular.

Then the case without pass-thru I'd call it vp.select because this does seem the obvious extension from the IR instruction select to VPred.

Agreed.

I understood from an earlier VPred sync call that you already had considered this scheme with two intrinsics and it had some drawbacks, right? Mind to elaborate? Perhaps a vectorizer will mostly always use vp.merge and never vp.select so the latter is less interesting to have?

The original scheme had a vp.merge intrinsic with pivot and no mask and vp.select as you'd expect it. That was designed only with VE in mind where a select will always retain the lanes above evl.
I mostly disliked the verbosity and hoped to fuse the functionality of both in one intrinsic.
There is no way around this now that we need a concise way to express select with evl.
Having a vp.merge(%m, %x, %y, %evl) that reains lanes above evl and vp.select(%m, %x, %y, %evl)that doesn't would be fine though.

[simoll]

• removed i1 mode parameter (will address this with another intrinsic, eg vp.merge, which retains lane above %evl(
• rebased

[frasercrmck]

Just nits and questions are this stage, really.

llvm/docs/LangRef.rst

17771

Nit: the select instruction is defined as having a "condition" whereas this is a "mask". There's a bit of mixing of "mask" and "condition" in the follow paragraphs too. Should we try to stick to "condition"?

llvm/include/llvm/IR/Intrinsics.td

1510

Nit: is this really a shuffle? I guess in a way it is, but it feels like "shuffles" and "selects" are already two distinct terms in the non-VP world.

[simoll]

llvm/docs/LangRef.rst

17771

+1 for consistency. The difference to regular select is that the condition/mask has to be a vector. How about "condition vector"?

llvm/include/llvm/IR/Intrinsics.td

1510

I think it's nice to group all shuffle intrinsics together (intrinsics that rearrange lanes). Actually, vp.splice and the (upcoming) vp.merge/thePivotThing should be in the shuffle category as well.
It's pretty inconsequential anyway as this understanding of "shuffle" only exists in these source code comments and does not appear in the docs.

[simoll]

Phrasing: mask --> condition vector

[rogfer01]

LGTM. This will enable using those in VPlan as a later followup of @vkmr patches.

[rogfer01]

llvm/include/llvm/IR/Intrinsics.td

1511

Deep apologies because I didn't realise this earlier: shouldn't this intrinsic be at least IntrNoMem? It was in the reference patch and it'll probably make a difference when you add VP_SELECT.

llvm/docs/LangRef.rst

 Optimization Hint
 ^^^^^^^^^^^^^^^^^
 
 Some targets, such as AVX512, do not support the %evl parameter in hardware.
 The use of an effective %evl is discouraged for those targets.  The function
 ``TargetTransformInfo::hasActiveVectorLength()`` returns true when the target
 has native support for %evl.
 
+.. _int_vp_select:
+
+'``llvm.vp.select.*``' Intrinsics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+      declare <16 x i32>  @llvm.vp.select.v16i32 (<16 x i1> <condition>, <16 x i32> <on_true>, <16 x i32> <on_false>, i32 <evl>)
+      declare <vscale x 4 x i64>  @llvm.vp.select.nxv4i64 (<vscale x 4 x i1> <condition>, <vscale x 4 x i32> <on_true>, <vscale x 4 x i32> <on_false>, i32 <evl>)
+
+Overview:
+"""""""""
+
+The '``llvm.vp.select``' intrinsic is used to choose one value based on a
+condition vector, without IR-level branching.
+
+Arguments:
+""""""""""
+
+The first operand is a vector of ``i1`` and indicates the condition.  The
+second operand is the value that is selected where the condition vector is
+true.  The third operand is the value that is selected where the condition
+vector is false.  The vectors must be of the same size.  The fourth operand is
+the explicit vector length.
+
+#. The optional ``fast-math flags`` marker indicates that the select has one or
+   more :ref:`fast-math flags <fastmath>`. These are optimization hints to
+   enable otherwise unsafe floating-point optimizations. Fast-math flags are
+   only valid for selects that return a floating-point scalar or vector type,
+   or an array (nested to any depth) of floating-point scalar or vector types.
+
+Semantics:
+""""""""""
+
+The intrinsic selects lanes from the second and third operand depending on a
+condition vector.
+
+All result lanes at positions greater or equal than ``%evl`` are undefined.
+For all lanes below ``%evl`` where the condition vector is true the lane is
+taken from the second operand.  Otherwise, the lane is taken from the third
+operand.
+
+Example:
+""""""""
+
+.. code-block:: llvm
+
+      %r = call <4 x i32> @llvm.vp.select.v4i32(<4 x i1> %cond, <4 x i32> %on_true, <4 x i32> %on_false, i32 %evl)
+
+      ;;; Expansion.
+      ;; Any result is legal on lanes at and above %evl.
+      %also.r = select <4 x i1> %cond, <4 x i32> %on_true, <4 x i32> %on_false
+
+
 
 .. _int_vp_add:
 
 '``llvm.vp.add.*``' Intrinsics
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 This is an overloaded intrinsic.
 
 ::
 
       declare <16 x i32>  @llvm.vp.add.v16i32 (<16 x i32> <left_op>, <16 x i32> <right_op>, <16 x i1> <mask>, i32 <vector_length>)
       declare <vscale x 4 x i32>  @llvm.vp.add.nxv4i32 (<vscale x 4 x i32> <left_op>, <vscale x 4 x i32> <right_op>, <vscale x 4 x i1> <mask>, i32 <vector_length>)
       declare <256 x i64>  @llvm.vp.add.v256i64 (<256 x i64> <left_op>, <256 x i64> <right_op>, <256 x i1> <mask>, i32 <vector_length>)
 
 Overview:
 """""""""
 
 Predicated integer addition of two vectors of integers.
 
 
 Arguments:
 """"""""""
 
 The first two operands and the result have the same vector of integer type. The
 third operand is the vector mask and has the same number of elements as the
 result vector type. The fourth operand is the explicit vector length of the
 operation.
 

llvm/include/llvm/IR/Intrinsics.td

def int_vp_fdiv : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
                                 LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
                                 llvm_i32_ty]>;
   def int_vp_frem : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
                               [ LLVMMatchType<0>,
                                 LLVMMatchType<0>,
                                 LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
                                 llvm_i32_ty]>;
 }
+// Shuffles.
+def int_vp_select : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
+                              [ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
+                                LLVMMatchType<0>,
+                                LLVMMatchType<0>,
+                                llvm_i32_ty]>;
 
 // Reductions
 let IntrProperties = [IntrSpeculatable, IntrNoMem, IntrNoSync, IntrWillReturn] in {
   def int_vp_reduce_fadd : DefaultAttrsIntrinsic<[LLVMVectorElementType<0>],
                                     [LLVMVectorElementType<0>,
                                      llvm_anyvector_ty,
                                      LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
                                      llvm_i32_ty]>;

llvm/include/llvm/IR/VPIntrinsics.def

 HELPER_REGISTER_REDUCTION_SEQ_VP(vp_reduce_fmul, VP_REDUCE_FMUL,
                                  VP_REDUCE_SEQ_FMUL,
                                  experimental_vector_reduce_fmul)
 
 #undef HELPER_REGISTER_REDUCTION_SEQ_VP
 
 ///// } Reduction
 
+///// Shuffles {
+
+// llvm.vp.select(mask,on_true,on_false,vlen)
+BEGIN_REGISTER_VP_INTRINSIC(vp_select, 0, 3)
+// BEGIN_REGISTER_VP_SDNODE(VP_SELECT, -1, vp_select, 0, 4)
+// END_REGISTER_CASES(vp_select, VP_SELECT)
+END_REGISTER_VP_INTRINSIC(vp_select)
+
+///// } Shuffles
+
 #undef BEGIN_REGISTER_VP
 #undef BEGIN_REGISTER_VP_INTRINSIC
 #undef BEGIN_REGISTER_VP_SDNODE
 #undef END_REGISTER_VP
 #undef END_REGISTER_VP_INTRINSIC
 #undef END_REGISTER_VP_SDNODE
 #undef HANDLE_VP_TO_OPC
 #undef HANDLE_VP_TO_CONSTRAINEDFP
 #undef HANDLE_VP_TO_INTRIN
 #undef HANDLE_VP_IS_MEMOP
 #undef HANDLE_VP_REDUCTION

llvm/lib/IR/IntrinsicInst.cpp

default: {
     Type *OverloadTy = Params[0]->getType();
     if (VPReductionIntrinsic::isVPReduction(VPID))
       OverloadTy =
           Params[*VPReductionIntrinsic::getVectorParamPos(VPID)]->getType();
 
     VPFunc = Intrinsic::getDeclaration(M, VPID, OverloadTy);
     break;
   }
+  case Intrinsic::vp_select:
+    VPFunc = Intrinsic::getDeclaration(M, VPID, {Params[1]->getType()});
+    break;
   case Intrinsic::vp_load:
     VPFunc = Intrinsic::getDeclaration(
         M, VPID,
         {Params[0]->getType()->getPointerElementType(), Params[0]->getType()});
     break;
   case Intrinsic::vp_gather:
     VPFunc = Intrinsic::getDeclaration(
         M, VPID,

llvm/unittests/IR/VPIntrinsicTest.cpp

std::unique_ptr<Module> createVPDeclarationModule() {
     for (const char *ReductionOpcode : ReductionIntOpcodes)
       Str << " declare i32 @llvm.vp.reduce." << ReductionOpcode
           << ".v8i32(i32, <8 x i32>, <8 x i1>, i32) ";
 
     for (const char *ReductionOpcode : ReductionFPOpcodes)
       Str << " declare float @llvm.vp.reduce." << ReductionOpcode
           << ".v8f32(float, <8 x float>, <8 x i1>, i32) ";
 
+    Str << " declare <8 x i32> @llvm.vp.select.v8i32(<8 x i1>, <8 x i32>, <8 x "
+           "i32>, i32)";
     return parseAssemblyString(Str.str(), Err, C);
   }
 };
 
 /// Check that the property scopes include/llvm/IR/VPIntrinsics.def are closed.
 TEST_F(VPIntrinsicTest, VPIntrinsicsDefScopes) {
   Optional<Intrinsic::ID> ScopeVPID;
 #define BEGIN_REGISTER_VP_INTRINSIC(VPID, ...)                                 \