Vector constrained FP intrinsics

[cameron.mcinally]

Summary

Let me preface this by noting that I'm not yet comfortable with the constrained FP intrinsics implementation, so I'm posting this patch to initiate a conversation on how I should proceed...

I wrote some tests to exercise the vector variants of the constrained FP intrinsics and found that some of the vector lib calls may need special handling. E.g. vector POW on X86 is expanded. This will require special handling of STRICT_FPOW in VectorLegalizer::LegalizeOp(...).

To handle STRICT_FPOW, I followed the existing code in SelectionDAGLegalize::LegalizeOp(...), since I believe this is in line with the intended implementation for the STRICT_XXX opcodes.

That being said, it may be more maintainable to reuse a helper function that converts a STRICT_XXX into its corresponding XXX node early in the LegailizeOp(...) functions. For example,

if (Node->isStrictFPOpcode())
  Node = DAG.mutateStrictFPToFP(Node);

Thoughts?

[andrew.w.kaylor]

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

312

Do you actually want to mutate the node at this point?

There's a function, getStrictFPOpcodeAction() that does something very close to what you are doing here except without mutating the node. Would moving that to an accessible location do what you need?

[cameron.mcinally]

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

312

Ah, I see the problem now. The splitting and expanding code needs to be taught how to handle the chain on constrained operations. I was inadvertently trying to mutate away the chain. This will probably take a while to sort out...

Thanks, Andrew.

[uweigand]

In addition to what Andrew said, there's another reason not to mutate this early: sooner or later, we'll need to give the target the chance to actually model strict operations differently (e.g. via more precise tracking of the FP status bit dependencies). At this point, the information which operations were strict must be preserved until the MI level anyway.

See e.g. my proposed patch here:
https://reviews.llvm.org/D45576

[cameron.mcinally]

In D46967#1107967, @uweigand wrote:

...

See e.g. my proposed patch here:
https://reviews.llvm.org/D45576

Oh, interesting. So the second comment on D45576 suggests that there's more design for this than I've seen so far. I'll close this Diff, if there are not objections. It's pretty clear that this is not the correct path going forward.

Ulrich, is there support I can provide you without stepping on your toes? Maybe I could write tests to exercise the intrinsics? Or something else?

[uweigand]

In D46967#1108432, @cameron.mcinally wrote:

Oh, interesting. So the second comment on D45576 suggests that there's more design for this than I've seen so far. I'll close this Diff, if there are not objections. It's pretty clear that this is not the correct path going forward.

On the other hand, the problem you've noticed here is of course still real, and needs to be addressed one way or the other. It looks like we will have to have support for properly legalizing STRICT_ operations on unsupported (vector) types. This isn't really addressed by my patch either. (I didn't even realize there was a problem; I had not looked a STRICT operations on vector types at all so far.)

Ulrich, is there support I can provide you without stepping on your toes? Maybe I could write tests to exercise the intrinsics? Or something else?

If you still want to go ahead and implement legalization along the lines of your previous comment ("The splitting and expanding code needs to be taught how to handle the chain on constrained operations."), that wouldn't be stepping on my toes at all :-)

[cameron.mcinally]

Hi Ulrich,

I have an updated patch ready to go to support expanding strict operations.

I also have a supporting patch waiting for review, namely D47380. I'm assuming that a standalone prerequisite patch is preferred to a blob of code all at once. Please correct me if I'm wrong...

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

for (SDNode::value_iterator J = Node->value_begin(), E = Node->value_end();
     HasVectorValue |= J->isVector();
   if (!HasVectorValue)
     return TranslateLegalizeResults(Op, Result);
 
   EVT QueryType;
   switch (Op.getOpcode()) {
   default:
     return TranslateLegalizeResults(Op, Result);
+  case ISD::STRICT_FSQRT:
+  case ISD::STRICT_FMA:
+  case ISD::STRICT_FPOW:
+  case ISD::STRICT_FPOWI:
+  case ISD::STRICT_FSIN:
+  case ISD::STRICT_FCOS:
+  case ISD::STRICT_FEXP:
+  case ISD::STRICT_FEXP2:
+  case ISD::STRICT_FLOG:
+  case ISD::STRICT_FLOG10:
+  case ISD::STRICT_FLOG2:
+  case ISD::STRICT_FRINT:
+  case ISD::STRICT_FNEARBYINT:
+    // These pseudo-ops get legalized as if they were their non-strict
+    // equivalent.  For instance, if ISD::FSQRT is legal then ISD::STRICT_FSQRT
+    // is also legal, but if ISD::FSQRT requires expansion then so does
+    // ISD::STRICT_FSQRT.
+    Node = DAG.mutateStrictFPToFP(Node);
+    QueryType = Node->getValueType(0);
+    break;
   case ISD::ADD:
   case ISD::SUB:
   case ISD::MUL:
   case ISD::SDIV:
   case ISD::UDIV:
   case ISD::SREM:
   case ISD::UREM:
   case ISD::SDIVREM:

test/CodeGen/X86/vector-constrained-fp-intrinsics.ll

+; RUN: llc -O3 -mtriple=x86_64-pc-linux < %s | FileCheck --check-prefix=COMMON --check-prefix=NO-FMA --check-prefix=FMACALL64 --check-prefix=FMACALL32 %s
+; RUN: llc -O3 -mtriple=x86_64-pc-linux -mattr=+fma < %s | FileCheck -check-prefix=COMMON --check-prefix=HAS-FMA --check-prefix=FMA64 --check-prefix=FMA32 %s
+
+; CHECK-LABEL: constrained_vector_fdiv
+; COMMON: divpd
+define <2 x double> @constrained_vector_fdiv() {
+entry:
+  %div = call <2 x double> @llvm.experimental.constrained.fdiv.v2f64(
+                                               <2 x double> <double 1.000000e+00, double 1.000000e+00>,
+                                               <2 x double> <double 1.000000e+01, double 1.000000e+01>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %div
+}
+
+; CHECK-LABEL: constrained_vector_fmul
+; COMMON: mulpd
+define <2 x double> @constrained_vector_fmul(<2 x double> %a) {
+entry:
+  %mul = call <2 x double> @llvm.experimental.constrained.fmul.v2f64(
+                                               <2 x double> <double 0x7FEFFFFFFFFFFFFF, double 0x7FEFFFFFFFFFFFFF>,
+                                               <2 x double> <double 2.000000e+00, double 2.000000e+00>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %mul
+}
+
+; CHECK-LABEL: constrained_vector_fadd
+; COMMON: addpd
+define <2 x double> @constrained_vector_fadd() {
+entry:
+  %add = call <2 x double> @llvm.experimental.constrained.fadd.v2f64(
+                                               <2 x double> <double 0x7FEFFFFFFFFFFFFF, double 0x7FEFFFFFFFFFFFFF>,
+                                               <2 x double> <double 1.000000e+00, double 1.000000e+00>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %add
+}
+
+; CHECK-LABEL: constrained_vector_fsub
+; COMMON: subpd
+define <2 x double> @constrained_vector_fsub() {
+entry:
+  %sub = call <2 x double> @llvm.experimental.constrained.fsub.v2f64(
+                                               <2 x double> <double 0xFFEFFFFFFFFFFFFF, double 0xFFEFFFFFFFFFFFFF>,
+                                               <2 x double> <double 1.000000e+00, double 1.000000e+00>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %sub
+}
+
+; CHECK-LABEL: constrained_vector_sqrt
+; COMMON: sqrtpd
+define <2 x double> @constrained_vector_sqrt() {
+entry:
+  %sqrt = call <2 x double> @llvm.experimental.constrained.sqrt.v2f64(
+                                               <2 x double> <double 42.0, double 42.0>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %sqrt
+}
+
+; CHECK-LABEL: constrained_vector_pow
+; COMMON: pow
+define <2 x double> @constrained_vector_pow() {
+entry:
+  %pow = call <2 x double> @llvm.experimental.constrained.pow.v2f64(
+                                               <2 x double> <double 42.1, double 42.1>,
+                                               <2 x double> <double 3.0, double 3.0>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %pow
+}
+
+; CHECK-LABEL: constrained_vector_sin
+; COMMON: sin
+define <2 x double> @constrained_vector_sin() {
+entry:
+  %sin = call <2 x double> @llvm.experimental.constrained.sin.v2f64(
+                                               <2 x double> <double 42.0, double 42.0>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %sin
+}
+
+; CHECK-LABEL: constrained_vector_cos
+; COMMON: cos
+define <2 x double> @constrained_vector_cos() {
+entry:
+  %cos = call <2 x double> @llvm.experimental.constrained.cos.v2f64(
+                                               <2 x double> <double 42.0, double 42.0>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %cos
+}
+
+; CHECK-LABEL: constrained_vector_exp
+; COMMON: exp
+define <2 x double> @constrained_vector_exp() {
+entry:
+  %exp = call <2 x double> @llvm.experimental.constrained.exp.v2f64(
+                                               <2 x double> <double 42.0, double 42.0>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %exp
+}
+
+; CHECK-LABEL: constrained_vector_exp2
+; COMMON: exp2
+define <2 x double> @constrained_vector_exp2() {
+entry:
+  %exp2 = call <2 x double> @llvm.experimental.constrained.exp2.v2f64(
+                                               <2 x double> <double 42.1, double 42.1>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %exp2
+}
+
+; CHECK-LABEL: constrained_vector_log
+; COMMON: log
+define <2 x double> @constrained_vector_log() {
+entry:
+  %log = call <2 x double> @llvm.experimental.constrained.log.v2f64(
+                                               <2 x double> <double 42.0, double 42.0>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %log
+}
+
+; CHECK-LABEL: constrained_vector_log10
+; COMMON: log10
+define <2 x double> @constrained_vector_log10() {
+entry:
+  %log10 = call <2 x double> @llvm.experimental.constrained.log10.v2f64(
+                                               <2 x double> <double 42.0, double 42.0>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %log10
+}
+
+; CHECK-LABEL: constrained_vector_log2
+; COMMON: log2
+define <2 x double> @constrained_vector_log2() {
+entry:
+  %log2 = call <2 x double> @llvm.experimental.constrained.log2.v2f64(
+                                               <2 x double> <double 42.0, double 42.0>,
+                                               metadata !"round.dynamic",
+                                               metadata !"fpexcept.strict")
+  ret <2 x double> %log2
+}
+
+declare <2 x double> @llvm.experimental.constrained.fdiv.v2f64(<2 x double>, <2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.fmul.v2f64(<2 x double>, <2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.fadd.v2f64(<2 x double>, <2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.fsub.v2f64(<2 x double>, <2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.sqrt.v2f64(<2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.pow.v2f64(<2 x double>, <2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.sin.v2f64(<2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.cos.v2f64(<2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.exp.v2f64(<2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.exp2.v2f64(<2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.log.v2f64(<2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.log10.v2f64(<2 x double>, metadata, metadata)
+declare <2 x double> @llvm.experimental.constrained.log2.v2f64(<2 x double>, metadata, metadata)
+