[VP] Binary floating-point intrinsics.

[simoll]

Summary

This patch implements vector-predicated intrinsics on IR level for fadd, fsub, fmul, fdiv and frem.
There operate in the default floating-point environment. We will use constrained fp operand bundles for constrained vector-predicated fp math (D93455).

[simoll]

llvm/include/llvm/IR/VPIntrinsics.def

174

.. fsub is missing here.. i will supplement it and add a unit test to check the integrity of the VPIntrinsics.def file.

[simoll]

• This patch is based on D93534 .
• Added missing vp.fsub.
• Added constrained fp mapping tests (whether the constrained intrinsic actually exists and has as many metadata params as the VPIntrinsic accepts). This is in preparation of D93455 .

[frasercrmck]

llvm/docs/LangRef.rst

17384

I realise this inherits from the documentation of the integer intrinsics, but I was wondering if this can be expressed as being equivalent to <%evl x float> in the following example. Would that be any clearer that the intrinsic is conceptually working on vectors of length %evl isn't actually executing the lanes above %evl (as in %t below)?

17408

addition -> subtraction

17422

same here: addition and maybe add <i_fsub>?

17457

addition -> multiplication

17471

addition -> multiplication

17506

addition -> division

17520

same here

17555

addition -> remainder

17569

addition -> remainder

llvm/include/llvm/IR/VPIntrinsics.def

162

integer -> floating-point?

[simoll]

llvm/docs/LangRef.rst

17384

I see what are you getting at with this. I am not sure that introducing novel syntax only for explaining things is really helpful. My take here is that most aren't used to the %evl concept - but it is not a complicated thing per se.

[simoll]

The unit test changes in this patch depend on the cleanup work done in D93534 . Please have a look at that one too!

[simoll]

• Fixed LangRef copy pasta bugs.
• Formatting in VPIntrinsics.def
• NFC

[simoll]

With D93534 out of the way, we should have cleared all hurdles for this one. Where do we stand, comments, LGTMs?

[frasercrmck]

llvm/unittests/IR/VPIntrinsicTest.cpp

22

Is this used?

154

Does this condition no longer hold?

[simoll]

llvm/unittests/IR/VPIntrinsicTest.cpp

41

This is why we need the sstream include.

83

This assertion is redundantly checked in the GetParamPos test.

[simoll]

Any input for this patch? Happy for people to step up for a LGTM ;-)

[majnemer]

llvm/docs/LangRef.rst

17387

Would it be more general/useful to have the intrinsics take an alternative value just like llvm.masked.load and its passthru operand? You would get identical behavior to your proposal by setting this passthru to undef.

[simoll]

llvm/docs/LangRef.rst

17387

My take is this: if there was a passthru parameter we'd still have to optimize/match the op+select into op_with_passthru. So, if the simple op+select idiom (that people are already using, btw) suffices to model passthru why add a redundant parameter/code path?

We had this discussion a while back: https://reviews.llvm.org/D57504#1851456

[majnemer]

llvm/docs/LangRef.rst

17387

Ah, I see. Fair enough :)

[simoll]

llvm/docs/LangRef.rst

17387

Thanks for following up on your drive-by comment ;-)

[craig.topper]

llvm/include/llvm/IR/IntrinsicInst.h

256

Should these be in D93455 so they aren't unused functions?

259

These names should be lower case per coding standards.

261

a exception -> an exception

llvm/unittests/IR/VPIntrinsicTest.cpp

22

This looks duplicated

[simoll]

Removed unused HasRoundingMode, etc functions.
Cleanup.

[simoll]

Make clang-format happy

[craig.topper]

LGTM

llvm/docs/LangRef.rst

.. code-block:: llvm
 
       %r = call <4 x i32> @llvm.vp.xor.v4i32(<4 x i32> %a, <4 x i32> %b, <4 x i1> %mask, i32 %evl)
       ;; For all lanes below %evl, %r is lane-wise equivalent to %also.r
 
       %t = xor <4 x i32> %a, %b
       %also.r = select <4 x i1> %mask, <4 x i32> %t, <4 x i32> undef
 
 
+.. _int_vp_fadd:
+
+'``llvm.vp.fadd.*``' Intrinsics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+      declare <16 x float>  @llvm.vp.fadd.v16f32 (<16 x float> <left_op>, <16 x float> <right_op>, <16 x i1> <mask>, i32 <vector_length>)
+      declare <vscale x 4 x float>  @llvm.vp.fadd.nxv4f32 (<vscale x 4 x float> <left_op>, <vscale x 4 x float> <right_op>, <vscale x 4 x i1> <mask>, i32 <vector_length>)
+      declare <256 x double>  @llvm.vp.fadd.v256f64 (<256 x double> <left_op>, <256 x double> <right_op>, <256 x i1> <mask>, i32 <vector_length>)
+
+Overview:
+"""""""""
+
+Predicated floating-point addition of two vectors of floating-point values.
+
+
+Arguments:
+""""""""""
+
+The first two operands and the result have the same vector of floating-point type. The
+third operand is the vector mask and has the same number of elements as the
+result vector type. The fourth operand is the explicit vector length of the
+operation.
+
+Semantics:
+""""""""""
+
+The '``llvm.vp.fadd``' intrinsic performs floating-point addition (:ref:`add <i_fadd>`)
+of the first and second vector operand on each enabled lane.  The result on
+disabled lanes is undefined.  The operation is performed in the default
+floating-point environment.
+
+Examples:
+"""""""""
+
+.. code-block:: llvm
+
+      %r = call <4 x float> @llvm.vp.fadd.v4f32(<4 x float> %a, <4 x float> %b, <4 x i1> %mask, i32 %evl)
+      ;; For all lanes below %evl, %r is lane-wise equivalent to %also.r
+
+      %t = fadd <4 x float> %a, %b
+      %also.r = select <4 x i1> %mask, <4 x float> %t, <4 x float> undef
+
+
+.. _int_vp_fsub:
+
+'``llvm.vp.fsub.*``' Intrinsics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+      declare <16 x float>  @llvm.vp.fsub.v16f32 (<16 x float> <left_op>, <16 x float> <right_op>, <16 x i1> <mask>, i32 <vector_length>)
+      declare <vscale x 4 x float>  @llvm.vp.fsub.nxv4f32 (<vscale x 4 x float> <left_op>, <vscale x 4 x float> <right_op>, <vscale x 4 x i1> <mask>, i32 <vector_length>)
+      declare <256 x double>  @llvm.vp.fsub.v256f64 (<256 x double> <left_op>, <256 x double> <right_op>, <256 x i1> <mask>, i32 <vector_length>)
+
+Overview:
+"""""""""
+
+Predicated floating-point addition of two vectors of floating-point values.
+
+
+Arguments:
+""""""""""
+
+The first two operands and the result have the same vector of floating-point type. The
+third operand is the vector mask and has the same number of elements as the
+result vector type. The fourth operand is the explicit vector length of the
+operation.
+
+Semantics:
+""""""""""
+
+The '``llvm.vp.fsub``' intrinsic performs floating-point addition (:ref:`add <i_fsub>`)
+of the first and second vector operand on each enabled lane.  The result on
+disabled lanes is undefined.  The operation is performed in the default
+floating-point environment.
+
+Examples:
+"""""""""
+
+.. code-block:: llvm
+
+      %r = call <4 x float> @llvm.vp.fsub.v4f32(<4 x float> %a, <4 x float> %b, <4 x i1> %mask, i32 %evl)
+      ;; For all lanes below %evl, %r is lane-wise equivalent to %also.r
+
+      %t = fsub <4 x float> %a, %b
+      %also.r = select <4 x i1> %mask, <4 x float> %t, <4 x float> undef
+
+
+.. _int_vp_fmul:
+
+'``llvm.vp.fmul.*``' Intrinsics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+      declare <16 x float>  @llvm.vp.fmul.v16f32 (<16 x float> <left_op>, <16 x float> <right_op>, <16 x i1> <mask>, i32 <vector_length>)
+      declare <vscale x 4 x float>  @llvm.vp.fmul.nxv4f32 (<vscale x 4 x float> <left_op>, <vscale x 4 x float> <right_op>, <vscale x 4 x i1> <mask>, i32 <vector_length>)
+      declare <256 x double>  @llvm.vp.fmul.v256f64 (<256 x double> <left_op>, <256 x double> <right_op>, <256 x i1> <mask>, i32 <vector_length>)
+
+Overview:
+"""""""""
+
+Predicated floating-point addition of two vectors of floating-point values.
+
+
+Arguments:
+""""""""""
+
+The first two operands and the result have the same vector of floating-point type. The
+third operand is the vector mask and has the same number of elements as the
+result vector type. The fourth operand is the explicit vector length of the
+operation.
+
+Semantics:
+""""""""""
+
+The '``llvm.vp.fmul``' intrinsic performs floating-point addition (:ref:`add <i_fmul>`)
+of the first and second vector operand on each enabled lane.  The result on
+disabled lanes is undefined.  The operation is performed in the default
+floating-point environment.
+
+Examples:
+"""""""""
+
+.. code-block:: llvm
+
+      %r = call <4 x float> @llvm.vp.fmul.v4f32(<4 x float> %a, <4 x float> %b, <4 x i1> %mask, i32 %evl)
+      ;; For all lanes below %evl, %r is lane-wise equivalent to %also.r
+
+      %t = fmul <4 x float> %a, %b
+      %also.r = select <4 x i1> %mask, <4 x float> %t, <4 x float> undef
+
+
+.. _int_vp_fdiv:
+
+'``llvm.vp.fdiv.*``' Intrinsics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+      declare <16 x float>  @llvm.vp.fdiv.v16f32 (<16 x float> <left_op>, <16 x float> <right_op>, <16 x i1> <mask>, i32 <vector_length>)
+      declare <vscale x 4 x float>  @llvm.vp.fdiv.nxv4f32 (<vscale x 4 x float> <left_op>, <vscale x 4 x float> <right_op>, <vscale x 4 x i1> <mask>, i32 <vector_length>)
+      declare <256 x double>  @llvm.vp.fdiv.v256f64 (<256 x double> <left_op>, <256 x double> <right_op>, <256 x i1> <mask>, i32 <vector_length>)
+
+Overview:
+"""""""""
+
+Predicated floating-point addition of two vectors of floating-point values.
+
+
+Arguments:
+""""""""""
+
+The first two operands and the result have the same vector of floating-point type. The
+third operand is the vector mask and has the same number of elements as the
+result vector type. The fourth operand is the explicit vector length of the
+operation.
+
+Semantics:
+""""""""""
+
+The '``llvm.vp.fdiv``' intrinsic performs floating-point addition (:ref:`add <i_fdiv>`)
+of the first and second vector operand on each enabled lane.  The result on
+disabled lanes is undefined.  The operation is performed in the default
+floating-point environment.
+
+Examples:
+"""""""""
+
+.. code-block:: llvm
+
+      %r = call <4 x float> @llvm.vp.fdiv.v4f32(<4 x float> %a, <4 x float> %b, <4 x i1> %mask, i32 %evl)
+      ;; For all lanes below %evl, %r is lane-wise equivalent to %also.r
+
+      %t = fdiv <4 x float> %a, %b
+      %also.r = select <4 x i1> %mask, <4 x float> %t, <4 x float> undef
+
+
+.. _int_vp_frem:
+
+'``llvm.vp.frem.*``' Intrinsics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic.
+
+::
+
+      declare <16 x float>  @llvm.vp.frem.v16f32 (<16 x float> <left_op>, <16 x float> <right_op>, <16 x i1> <mask>, i32 <vector_length>)
+      declare <vscale x 4 x float>  @llvm.vp.frem.nxv4f32 (<vscale x 4 x float> <left_op>, <vscale x 4 x float> <right_op>, <vscale x 4 x i1> <mask>, i32 <vector_length>)
+      declare <256 x double>  @llvm.vp.frem.v256f64 (<256 x double> <left_op>, <256 x double> <right_op>, <256 x i1> <mask>, i32 <vector_length>)
+
+Overview:
+"""""""""
+
+Predicated floating-point addition of two vectors of floating-point values.
+
+
+Arguments:
+""""""""""
+
+The first two operands and the result have the same vector of floating-point type. The
+third operand is the vector mask and has the same number of elements as the
+result vector type. The fourth operand is the explicit vector length of the
+operation.
+
+Semantics:
+""""""""""
+
+The '``llvm.vp.frem``' intrinsic performs floating-point addition (:ref:`add <i_frem>`)
+of the first and second vector operand on each enabled lane.  The result on
+disabled lanes is undefined.  The operation is performed in the default
+floating-point environment.
+
+Examples:
+"""""""""
+
+.. code-block:: llvm
+
+      %r = call <4 x float> @llvm.vp.frem.v4f32(<4 x float> %a, <4 x float> %b, <4 x i1> %mask, i32 %evl)
+      ;; For all lanes below %evl, %r is lane-wise equivalent to %also.r
+
+      %t = frem <4 x float> %a, %b
+      %also.r = select <4 x i1> %mask, <4 x float> %t, <4 x float> undef
+
+
+
 .. _int_get_active_lane_mask:
 
 '``llvm.get.active.lane.mask.*``' Intrinsics
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 This is an overloaded intrinsic.

llvm/include/llvm/IR/IntrinsicInst.h

public:
   static Optional<int> GetVectorLengthParamPos(Intrinsic::ID IntrinsicID);
 
   /// The llvm.vp.* intrinsics for this instruction Opcode
   static Intrinsic::ID GetForOpcode(unsigned OC);
 
   // Whether \p ID is a VP intrinsic ID.
   static bool IsVPIntrinsic(Intrinsic::ID);
 
+  /// Constrained FP {
+
+  // Whether the VP intrinsic \p ID can have a rounding mode bundle.
+  static bool HasRoundingMode(Intrinsic::ID);
+
+  // Whether the VP intrinsic \p ID can have a exception behavior
+  // bundle.
+  static bool HasExceptionBehavior(Intrinsic::ID);
+
+  /// } Constrained FP
+
   /// \return the mask parameter or nullptr.
   Value *getMaskParam() const;
 
   /// \return the vector length parameter or nullptr.
   Value *getVectorLengthParam() const;
 
   /// \return whether the vector length param can be ignored.
   bool canIgnoreVectorLengthParam() const;

llvm/include/llvm/IR/Intrinsics.td

def int_vp_srem : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
                                 llvm_i32_ty]>;
   def int_vp_urem : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
                               [ LLVMMatchType<0>,
                                 LLVMMatchType<0>,
                                 LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
                                 llvm_i32_ty]>;
 }
 
+// Floating-point arithmetic.
+let IntrProperties =
+    [IntrSpeculatable, IntrNoMem, IntrNoSync, IntrWillReturn] in {
+  def int_vp_fadd : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
+                             [ LLVMMatchType<0>,
+                               LLVMMatchType<0>,
+                               LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
+                               llvm_i32_ty]>;
+  def int_vp_fsub : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
+                             [ LLVMMatchType<0>,
+                               LLVMMatchType<0>,
+                               LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
+                               llvm_i32_ty]>;
+  def int_vp_fmul  : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
+                              [ LLVMMatchType<0>,
+                                LLVMMatchType<0>,
+                                LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
+                                llvm_i32_ty]>;
+  def int_vp_fdiv : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
+                              [ LLVMMatchType<0>,
+                                LLVMMatchType<0>,
+                                LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
+                                llvm_i32_ty]>;
+  def int_vp_frem : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
+                              [ LLVMMatchType<0>,
+                                LLVMMatchType<0>,
+                                LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
+                                llvm_i32_ty]>;
+}
+
 def int_get_active_lane_mask:
   DefaultAttrsIntrinsic<[llvm_anyvector_ty],
             [llvm_anyint_ty, LLVMMatchType<1>],
             [IntrNoMem, IntrNoSync, IntrWillReturn]>;
 
 //===-------------------------- Masked Intrinsics -------------------------===//
 //
 def int_masked_load:

llvm/include/llvm/IR/VPIntrinsics.def

 // Property Macros {
 
 // The intrinsic and/or SDNode has the same function as this LLVM IR Opcode.
 // \p OPC  The standard IR opcode.
 #ifndef HANDLE_VP_TO_OPC
 #define HANDLE_VP_TO_OPC(OPC)
 #endif
 
+// Whether the intrinsic may have a rounding mode or exception behavior operand
+// bundle.
+// \p HASROUND   '1' if the intrinsic can have a rounding mode operand bundle,
+//               '0' otherwise.
+// \p HASEXCEPT  '1' if the intrinsic can have an exception behavior operand
+//               bundle, '0' otherwise.
+// \p INTRINID  The constrained fp intrinsic this VP intrinsic corresponds to.
+#ifndef HANDLE_VP_TO_CONSTRAINEDFP
+#define HANDLE_VP_TO_CONSTRAINEDFP(HASROUND, HASEXCEPT, INTRINID)
+#endif
+
 /// } Property Macros
 
 ///// Integer Arithmetic {
 
 // Specialized helper macro for integer binary operators (%x, %y, %mask, %evl).
 #ifdef HELPER_REGISTER_BINARY_INT_VP
 #error "The internal helper macro HELPER_REGISTER_BINARY_INT_VP is already defined!"
 #endif

 
 // llvm.vp.xor(x,y,mask,vlen)
 HELPER_REGISTER_BINARY_INT_VP(vp_xor, VP_XOR, Xor)
 
 #undef HELPER_REGISTER_BINARY_INT_VP
 
 ///// } Integer Arithmetic
 
+///// Floating-Point Arithmetic {
+
+// Specialized helper macro for integer binary operators (%x, %y, %mask, %evl).
+#ifdef HELPER_REGISTER_BINARY_FP_VP
+#error "The internal helper macro HELPER_REGISTER_BINARY_FP_VP is already defined!"
+#endif
+#define HELPER_REGISTER_BINARY_FP_VP(OPSUFFIX, SDOPC, OPC) \
+BEGIN_REGISTER_VP(vp_##OPSUFFIX, 2, 3, SDOPC, -1) \
+HANDLE_VP_TO_OPC(OPC) \
+HANDLE_VP_TO_CONSTRAINEDFP(1,1,experimental_constrained_##OPSUFFIX) \
+END_REGISTER_VP(vp_##OPSUFFIX, SDOPC)
+
+// llvm.vp.fadd(x,y,mask,vlen)
+HELPER_REGISTER_BINARY_FP_VP(fadd, VP_FADD, FAdd)
+
+// llvm.vp.fsub(x,y,mask,vlen)
+HELPER_REGISTER_BINARY_FP_VP(fsub, VP_FSUB, FSub)
+
+// llvm.vp.fmul(x,y,mask,vlen)
+HELPER_REGISTER_BINARY_FP_VP(fmul, VP_FMUL, FMul)
+
+// llvm.vp.fdiv(x,y,mask,vlen)
+HELPER_REGISTER_BINARY_FP_VP(fdiv, VP_FDIV, FDiv)
+
+// llvm.vp.frem(x,y,mask,vlen)
+HELPER_REGISTER_BINARY_FP_VP(frem, VP_FREM, FRem)
+
+#undef HELPER_REGISTER_BINARY_FP_VP
+
+///// } Floating-Point Arithmetic
 
 #undef BEGIN_REGISTER_VP
 #undef BEGIN_REGISTER_VP_INTRINSIC
 #undef BEGIN_REGISTER_VP_SDNODE
 #undef END_REGISTER_VP
 #undef END_REGISTER_VP_INTRINSIC
 #undef END_REGISTER_VP_SDNODE
 #undef HANDLE_VP_TO_OPC
+#undef HANDLE_VP_TO_CONSTRAINEDFP

llvm/lib/IR/IntrinsicInst.cpp

 #define BEGIN_REGISTER_VP_INTRINSIC(VPID, MASKPOS, VLENPOS)                    \
   case Intrinsic::VPID:                                                        \
     break;
 #include "llvm/IR/VPIntrinsics.def"
   }
   return true;
 }
 
+bool VPIntrinsic::HasRoundingMode(Intrinsic::ID ID) {
+  switch (ID) {
+  default:
+    return false;
+
+#define BEGIN_REGISTER_VP_INTRINSIC(VPID, MASKPOS, VLENPOS)                    \
+  case Intrinsic::VPID:
+#define END_REGISTER_VP_INTRINSIC(VPID) return false;
+#define HANDLE_VP_TO_CONSTRAINEDFP(HASROUND, ...) return (bool)HASROUND;
+#include "llvm/IR/VPIntrinsics.def"
+  }
+}
+
+bool VPIntrinsic::HasExceptionBehavior(Intrinsic::ID ID) {
+  switch (ID) {
+  default:
+    return false;
+
+#define BEGIN_REGISTER_VP_INTRINSIC(VPID, MASKPOS, VLENPOS)                    \
+  case Intrinsic::VPID:
+#define END_REGISTER_VP_INTRINSIC(VPID) return false;
+#define HANDLE_VP_TO_CONSTRAINEDFP(HASROUND, HASEXCEPT, ...)                   \
+  return (bool)HASEXCEPT;
+#include "llvm/IR/VPIntrinsics.def"
+  }
+}
+
 // Equivalent non-predicated opcode
 unsigned VPIntrinsic::GetFunctionalOpcodeForVP(Intrinsic::ID ID) {
   unsigned FunctionalOC = Instruction::Call;
   switch (ID) {
   default:
     break;
 #define BEGIN_REGISTER_VP_INTRINSIC(VPID, ...) case Intrinsic::VPID:
 #define HANDLE_VP_TO_OPC(OPC) FunctionalOC = Instruction::OPC;

llvm/test/Verifier/vp-intrinsics.ll

define void @test_vp_int(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n) {
   %r8 = call <8 x i32> @llvm.vp.or.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
   %r9 = call <8 x i32> @llvm.vp.xor.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
   %rA = call <8 x i32> @llvm.vp.ashr.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n) 
   %rB = call <8 x i32> @llvm.vp.lshr.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n) 
   %rC = call <8 x i32> @llvm.vp.shl.v8i32(<8 x i32> %i0, <8 x i32> %i1, <8 x i1> %m, i32 %n)
   ret void
 }
 
+
+define void @test_vp_fp(<8 x double> %f0, <8 x double> %f1, <8 x i1> %m, i32 %n) {
+  %r0 = call <8 x double> @llvm.vp.fadd.v8f64(<8 x double> %f0, <8 x double> %f1, <8 x i1> %m, i32 %n)
+  %r1 = call <8 x double> @llvm.vp.fsub.v8f64(<8 x double> %f0, <8 x double> %f1, <8 x i1> %m, i32 %n)
+  %r2 = call <8 x double> @llvm.vp.fmul.v8f64(<8 x double> %f0, <8 x double> %f1, <8 x i1> %m, i32 %n)
+  %r3 = call <8 x double> @llvm.vp.fdiv.v8f64(<8 x double> %f0, <8 x double> %f1, <8 x i1> %m, i32 %n)
+  %r4 = call <8 x double> @llvm.vp.frem.v8f64(<8 x double> %f0, <8 x double> %f1, <8 x i1> %m, i32 %n)
+  ret void
+}
+
+; TODO: test_vp_constrained_fp
+
 ; integer arith
 declare <8 x i32> @llvm.vp.add.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.sub.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.mul.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.sdiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.srem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.udiv.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.urem.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 ; bit arith
 declare <8 x i32> @llvm.vp.and.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.or.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.xor.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
 declare <8 x i32> @llvm.vp.ashr.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) 
 declare <8 x i32> @llvm.vp.lshr.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) 
 declare <8 x i32> @llvm.vp.shl.v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32)
+; fp arith
+declare <8 x double> @llvm.vp.fadd.v8f64(<8 x double>, <8 x double>, <8 x i1>, i32)
+declare <8 x double> @llvm.vp.fsub.v8f64(<8 x double>, <8 x double>, <8 x i1>, i32)
+declare <8 x double> @llvm.vp.fmul.v8f64(<8 x double>, <8 x double>, <8 x i1>, i32)
+declare <8 x double> @llvm.vp.fdiv.v8f64(<8 x double>, <8 x double>, <8 x i1>, i32)
+declare <8 x double> @llvm.vp.frem.v8f64(<8 x double>, <8 x double>, <8 x i1>, i32)

llvm/unittests/IR/VPIntrinsicTest.cpp

 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/Verifier.h"
 #include "llvm/Support/SourceMgr.h"
 #include "gtest/gtest.h"
 #include <sstream>
 
+#include <sstream>
+
 using namespace llvm;
 
 namespace {
 
 class VPIntrinsicTest : public testing::Test {
 protected:
   LLVMContext Context;
 
   VPIntrinsicTest() : Context() {}
 
   LLVMContext C;
   SMDiagnostic Err;
 
   std::unique_ptr<Module> CreateVPDeclarationModule() {
     const char *BinaryIntOpcodes[] = {"add",  "sub",  "mul", "sdiv", "srem",
                                       "udiv", "urem", "and", "xor",  "or",
                                       "ashr", "lshr", "shl"};
     std::stringstream Str;
     for (const char *BinaryIntOpcode : BinaryIntOpcodes)
       Str << " declare <8 x i32> @llvm.vp." << BinaryIntOpcode
           << ".v8i32(<8 x i32>, <8 x i32>, <8 x i1>, i32) ";
 
+    const char *BinaryFPOpcodes[] = {"fadd", "fsub", "fmul", "fdiv", "frem"};
+    for (const char *BinaryFPOpcode : BinaryFPOpcodes)
+      Str << " declare <8 x float> @llvm.vp." << BinaryFPOpcode
+          << ".v8f32(<8 x float>, <8 x float>, <8 x i1>, i32) ";
+
     return parseAssemblyString(Str.str(), Err, C);
   }
 };
 
 /// Check that the property scopes include/llvm/IR/VPIntrinsics.def are closed.
 TEST_F(VPIntrinsicTest, VPIntrinsicsDefScopes) {
   Optional<Intrinsic::ID> ScopeVPID;
 #define BEGIN_REGISTER_VP_INTRINSIC(VPID, ...)                                 \

 TEST_F(VPIntrinsicTest, VPModuleComplete) {
   std::unique_ptr<Module> M = CreateVPDeclarationModule();
   assert(M);
 
   // Check that all @llvm.vp.* functions in the module are recognized vp
   // intrinsics.
   std::set<Intrinsic::ID> SeenIDs;
   for (const auto &VPDecl : *M) {
     ASSERT_TRUE(VPDecl.isIntrinsic());
     ASSERT_TRUE(VPIntrinsic::IsVPIntrinsic(VPDecl.getIntrinsicID()));
     SeenIDs.insert(VPDecl.getIntrinsicID());
   }
 
   // Check that every registered VP intrinsic has an instance in the test
   // module.
 #define BEGIN_REGISTER_VP_INTRINSIC(VPID, ...)                                 \
   ASSERT_TRUE(SeenIDs.count(Intrinsic::VPID));

 "  ret void "
 "}",
           Err, C);
 
   auto *F = M->getFunction("test_static_vlen");
   assert(F);
 
   const int NumExpected = 12;
-  const bool Expected[] = {false, true, false, false, false, true, false, false, true, false, true, false};
+  const bool Expected[] = {false, true,  false, false, false, true,
+                           false, false, true,  false, true,  false};
   int i = 0;
   for (auto &I : F->getEntryBlock()) {
     VPIntrinsic *VPI = dyn_cast<VPIntrinsic>(&I);
     if (!VPI)
       continue;
 
     ASSERT_LT(i, NumExpected);
     ASSERT_EQ(Expected[i], VPI->canIgnoreVectorLengthParam());
     ++i;
   }
 }
 
 /// Check that the argument returned by
 /// VPIntrinsic::Get<X>ParamPos(Intrinsic::ID) has the expected type.
 TEST_F(VPIntrinsicTest, GetParamPos) {
   std::unique_ptr<Module> M = CreateVPDeclarationModule();
   assert(M);
 
   for (Function &F : *M) {
-    ASSERT_TRUE(F.isIntrinsic());
     Optional<int> MaskParamPos =
         VPIntrinsic::GetMaskParamPos(F.getIntrinsicID());
     if (MaskParamPos.hasValue()) {
       Type *MaskParamType = F.getArg(MaskParamPos.getValue())->getType();
       ASSERT_TRUE(MaskParamType->isVectorTy());
-      ASSERT_TRUE(cast<VectorType>(MaskParamType)->getElementType()->isIntegerTy(1));
+      ASSERT_TRUE(
+          cast<VectorType>(MaskParamType)->getElementType()->isIntegerTy(1));
     }
 
     Optional<int> VecLenParamPos =
         VPIntrinsic::GetVectorLengthParamPos(F.getIntrinsicID());
     if (VecLenParamPos.hasValue()) {
       Type *VecLenParamType = F.getArg(VecLenParamPos.getValue())->getType();
       ASSERT_TRUE(VecLenParamType->isIntegerTy(32));
     }
for (const auto &VPDecl : *M) {
     Intrinsic::ID RoundTripVPID = VPIntrinsic::GetForOpcode(OPC);
 
     ASSERT_EQ(RoundTripVPID, VPID);
     ++FullTripCounts;
   }
   ASSERT_NE(FullTripCounts, 0u);
 }
 
+/// Check that the HANDLE_VP_TO_CONSTRAINEDFP maps to an existing intrinsic with
+/// the right amount of metadata args.
+TEST_F(VPIntrinsicTest, HandleToConstrainedFP) {
+#define HANDLE_VP_TO_CONSTRAINEDFP(HASROUND, HASEXCEPT, CFPID)                 \
+  {                                                                            \
+    SmallVector<Intrinsic::IITDescriptor, 5> T;                                \
+    Intrinsic::getIntrinsicInfoTableEntries(Intrinsic::CFPID, T);              \
+    unsigned NumMetadataArgs = 0;                                              \
+    for (auto TD : T)                                                          \
+      NumMetadataArgs += (TD.Kind == Intrinsic::IITDescriptor::Metadata);      \
+    ASSERT_EQ(NumMetadataArgs, HASROUND + HASEXCEPT);                          \
+  }
+#include "llvm/IR/VPIntrinsics.def"
+}
+
 } // end anonymous namespace