[AArch64] Properly validate the reciprocal estimation

[evandro]

Summary

Add check for legal data types when expanding into a Newton series.

[t.p.northover]

I think you forgot to add llvm-commits as a subscriber (that's what makes it get sent to the list, not the repository). Also tests.

[evandro]

@t.p.northover,

WRT the tests, I'd appreciate some help. In some cases, lowering would fail because the square root was being expanded into the series for MVT::v4f64, which is obviously not supported by AArch64. I'm at a loss as to why such data type, unsupported by the target, percolated through to the backend.

Would concocting the bytecode using such data type be a sensible test?

On another note, as I'm not very familiar with the latest versions of ARMv8, should FP16 be considered here?

Thank you.

[t.p.northover]

Would concocting the bytecode using such data type be a sensible test?

That's definitely a valid way to write the test if you can get it to crash like that. We should be accepting and legalizing all IR.

On another note, as I'm not very familiar with the latest versions of ARMv8, should FP16 be considered here?

Yes. f16, v4f16 and v8f16 are valid for both if Subtarget->hasFullFP16().

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

4579 ↗

(On Diff #63685)

Why the extra brackets?

[t.p.northover]

Also, probably v1i64 (a bit of a silly type, but it does exist and should work).

[t.p.northover]

Sorry, v1f64.

[evandro]

In D22267#481792, @t.p.northover wrote:

Also, probably v1i64 (a bit of a silly type, but it does exist and should work).

Whaaat? Please, explain.

[evandro]

In D22267#481801, @t.p.northover wrote:

Sorry, v1f64.

Ah, OK. How about v1f32?

[evandro]

@t.p.northover,

Please, correct me if I'm wrong, but there's neither FRECPE nor FRSQRTE insts in ARMv8.1, are there?

[t.p.northover]

Ah, OK. How about v1f32?

I wouldn't worry about that, I don't think it's used nearly as frequently. The v1f64 issue comes up because that's how arm_neon.h represents some vectors.

Please, correct me if I'm wrong, but there's neither FRECPE nor FRSQRTE insts in ARMv8.1, are there?

That's right. Full support for _Float16 is part of v8.2 I believe, but tracked as a separate feature in LLVM: "fullfp16" if you're doing -mattr.

[evandro]

In D22267#482029, @t.p.northover wrote:

That's right. Full support for _Float16 is part of v8.2 I believe, but tracked as a separate feature in LLVM: "fullfp16" if you're doing -mattr.

I don't have the v8.2 documentation at hand, so, please indulge me, does it support both FRECPE and FRSQRTE for _Float16?

[t.p.northover]

I don't have the v8.2 documentation at hand, so, please indulge me, does it support both FRECPE and FRSQRTE for _Float16?

It does:

$ echo "frecpe v0.4h, v1.4h" | llvm-mc -triple aarch64 -mattr=+fullfp16 -show-inst
frecpe v0.4h, v1.4h           // <MCInst #859 FRECPEv4f16
                              //  <MCOperand Reg:40>
                              //  <MCOperand Reg:41>>
$ echo "frsqrte v0.4h, v1.4h" | llvm-mc -triple aarch64 -mattr=+fullfp16 -show-inst
frsqrte	v0.4h, v1.4h            // <MCInst #934 FRSQRTEv4f16
                                //  <MCOperand Reg:40>
                                //  <MCOperand Reg:41>>

(plus similar results for scalar operations).

[evandro]

Thanks, @t.p.northover.

[evandro]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

4579 ↗

(On Diff #63685)

Oops!

[evandro]

Add validation for the _Float16 types.

[evandro]

Update existing test cases to include the error vector found in the wild.

[t.p.northover]

Thanks Evandro! I think this looks fine now.

llvm/trunk/lib/Target/AArch64/AArch64ISelLowering.cpp

 /// or the reciprocal square root.
 static SDValue getEstimate(const AArch64Subtarget &ST,
   const AArch64TargetLowering::DAGCombinerInfo &DCI, unsigned Opcode,
   const SDValue &Operand, unsigned &ExtraSteps) {
   if (!ST.hasNEON())
     return SDValue();
 
   EVT VT = Operand.getValueType();
+  if (VT != MVT::f64 && VT != MVT::v1f64 && VT != MVT::v2f64 &&
+      VT != MVT::f32 && VT != MVT::v1f32 &&
+      VT != MVT::v2f32 && VT != MVT::v4f32 &&
+      (!ST.hasFullFP16() ||
+       (VT != MVT::f16 && VT != MVT::v4f16 && VT != MVT::v8f16)))
+    return SDValue();
 
   std::string RecipOp;
   RecipOp = Opcode == (AArch64ISD::FRECPE) ? "div": "sqrt";
   RecipOp = ((VT.isVector()) ? "vec-": "") + RecipOp;
   RecipOp += (VT.getScalarType() == MVT::f64) ? "d": "f";
 
   TargetRecip Recips = DCI.DAG.getTarget().Options.Reciprocals;
   if (!Recips.isEnabled(RecipOp))

llvm/trunk/test/CodeGen/AArch64/recp-fastmath.ll

 ; FAULT-NEXT: fdiv
 
 ; CHECK-LABEL: f4recp:
 ; CHECK-NEXT: BB#0
 ; CHECK-NEXT: fmov
 ; CHECK-NEXT: frecpe
 }
 
+define <8 x float> @f8recp(<8 x float> %x) #0 {
+  %div = fdiv fast <8 x float> <float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0>, %x
+  ret <8 x float> %div
+
+; FAULT-LABEL: f8recp:
+; FAULT-NEXT: BB#0
+; FAULT-NEXT: fmov
+; FAULT-NEXT: fdiv
+; FAULT-NEXT: fdiv
+
+; CHECK-LABEL: f8recp:
+; CHECK-NEXT: BB#0
+; CHECK-NEXT: fmov
+; CHECK-NEXT: frecpe
+; CHECK: frecpe
+}
+
 define double @drecp(double %x) #0 {
   %div = fdiv fast double 1.0, %x
   ret double %div
 
 ; FAULT-LABEL: drecp:
 ; FAULT-NEXT: BB#0
 ; FAULT-NEXT: fmov
 ; FAULT-NEXT: fdiv

 ; FAULT-NEXT: fdiv
 
 ; CHECK-LABEL: d2recp:
 ; CHECK-NEXT: BB#0
 ; CHECK-NEXT: fmov
 ; CHECK-NEXT: frecpe
 }
 
+define <4 x double> @d4recp(<4 x double> %x) #0 {
+  %div = fdiv fast <4 x double> <double 1.0, double 1.0, double 1.0, double 1.0>, %x
+  ret <4 x double> %div
+
+; FAULT-LABEL: d4recp:
+; FAULT-NEXT: BB#0
+; FAULT-NEXT: fmov
+; FAULT-NEXT: fdiv
+; FAULT-NEXT: fdiv
+
+; CHECK-LABEL: d4recp:
+; CHECK-NEXT: BB#0
+; CHECK-NEXT: fmov
+; CHECK-NEXT: frecpe
+; CHECK: frecpe
+}
+
 attributes #0 = { nounwind "unsafe-fp-math"="true" }

llvm/trunk/test/CodeGen/AArch64/sqrt-fastmath.ll

-; RUN: llc < %s -mtriple=aarch64 -mattr=neon -recip=!sqrt,!vec-sqrt | FileCheck %s --check-prefix=FAULT
-; RUN: llc < %s -mtriple=aarch64 -mattr=neon -recip=sqrt,vec-sqrt   | FileCheck %s
-; RUN: llc < %s -mtriple=aarch64 -mattr=neon,-use-reverse-square-root  | FileCheck %s --check-prefix=FAULT
+; RUN: llc < %s -mtriple=aarch64 -mattr=neon -recip=!sqrt,!vec-sqrt   | FileCheck %s --check-prefix=FAULT
+; RUN: llc < %s -mtriple=aarch64 -mattr=neon,-use-reverse-square-root | FileCheck %s --check-prefix=FAULT
+; RUN: llc < %s -mtriple=aarch64 -mattr=neon -recip=sqrt,vec-sqrt     | FileCheck %s
 ; RUN: llc < %s -mtriple=aarch64 -mattr=neon,+use-reverse-square-root | FileCheck %s
 
 declare float @llvm.sqrt.f32(float) #1
-declare double @llvm.sqrt.f64(double) #1
 declare <2 x float> @llvm.sqrt.v2f32(<2 x float>) #1
 declare <4 x float> @llvm.sqrt.v4f32(<4 x float>) #1
+declare <8 x float> @llvm.sqrt.v8f32(<8 x float>) #1
+declare double @llvm.sqrt.f64(double) #1
 declare <2 x double> @llvm.sqrt.v2f64(<2 x double>) #1
+declare <4 x double> @llvm.sqrt.v4f64(<4 x double>) #1
 
 define float @fsqrt(float %a) #0 {
   %1 = tail call fast float @llvm.sqrt.f32(float %a)
   ret float %1
 
 ; FAULT-LABEL: fsqrt:
 ; FAULT-NEXT: BB#0
 ; FAULT-NEXT: fsqrt

 
 ; CHECK-LABEL: f4sqrt:
 ; CHECK-NEXT: BB#0
 ; CHECK-NEXT: fmov
 ; CHECK-NEXT: mov
 ; CHECK-NEXT: frsqrte
 }
 
+define <8 x float> @f8sqrt(<8 x float> %a) #0 {
+  %1 = tail call fast <8 x float> @llvm.sqrt.v8f32(<8 x float> %a) #2
+  ret <8 x float> %1
+
+; FAULT-LABEL: f8sqrt:
+; FAULT-NEXT: BB#0
+; FAULT-NEXT: fsqrt
+; FAULT-NEXT: fsqrt
+
+; CHECK-LABEL: f8sqrt:
+; CHECK-NEXT: BB#0
+; CHECK-NEXT: fmov
+; CHECK-NEXT: mov
+; CHECK-NEXT: frsqrte
+; CHECK: frsqrte
+}
+
 define double @dsqrt(double %a) #0 {
   %1 = tail call fast double @llvm.sqrt.f64(double %a)
   ret double %1
 
 ; FAULT-LABEL: dsqrt:
 ; FAULT-NEXT: BB#0
 ; FAULT-NEXT: fsqrt
 

 
 ; CHECK-LABEL: d2sqrt:
 ; CHECK-NEXT: BB#0
 ; CHECK-NEXT: fmov
 ; CHECK-NEXT: mov
 ; CHECK-NEXT: frsqrte
 }
 
+define <4 x double> @d4sqrt(<4 x double> %a) #0 {
+  %1 = tail call fast <4 x double> @llvm.sqrt.v4f64(<4 x double> %a) #4
+  ret <4 x double> %1
+
+; FAULT-LABEL: d4sqrt:
+; FAULT-NEXT: BB#0
+; FAULT-NEXT: fsqrt
+; FAULT-NEXT: fsqrt
+
+; CHECK-LABEL: d4sqrt:
+; CHECK-NEXT: BB#0
+; CHECK-NEXT: fmov
+; CHECK-NEXT: mov
+; CHECK-NEXT: frsqrte
+; CHECK: frsqrte
+}
+
 define float @frsqrt(float %a) #0 {
   %1 = tail call fast float @llvm.sqrt.f32(float %a)
   %2 = fdiv fast float 1.000000e+00, %1
   ret float %2
 
 ; FAULT-LABEL: frsqrt:
 ; FAULT-NEXT: BB#0
 ; FAULT-NEXT: fsqrt

 ; FAULT-NEXT: fsqrt
 
 ; CHECK-LABEL: f4rsqrt:
 ; CHECK-NEXT: BB#0
 ; CHECK-NEXT: fmov
 ; CHECK-NEXT: frsqrte
 }
 
+define <8 x float> @f8rsqrt(<8 x float> %a) #0 {
+  %1 = tail call fast <8 x float> @llvm.sqrt.v8f32(<8 x float> %a) #2
+  %2 = fdiv fast <8 x float> <float 1.000000e+00, float 1.000000e+00, float 1.000000e+00, float 1.000000e+00, float 1.000000e+00, float 1.000000e+00, float 1.000000e+00, float 1.000000e+00>, %1
+  ret <8 x float> %2
+
+; FAULT-LABEL: f8rsqrt:
+; FAULT-NEXT: BB#0
+; FAULT-NEXT: fsqrt
+; FAULT-NEXT: fsqrt
+
+; CHECK-LABEL: f8rsqrt:
+; CHECK-NEXT: BB#0
+; CHECK-NEXT: fmov
+; CHECK-NEXT: frsqrte
+; CHECK: frsqrte
+}
+
 define double @drsqrt(double %a) #0 {
   %1 = tail call fast double @llvm.sqrt.f64(double %a)
   %2 = fdiv fast double 1.000000e+00, %1
   ret double %2
 
 ; FAULT-LABEL: drsqrt:
 ; FAULT-NEXT: BB#0
 ; FAULT-NEXT: fsqrt

 ; FAULT-NEXT: fsqrt
 
 ; CHECK-LABEL: d2rsqrt:
 ; CHECK-NEXT: BB#0
 ; CHECK-NEXT: fmov
 ; CHECK-NEXT: frsqrte
 }
 
+define <4 x double> @d4rsqrt(<4 x double> %a) #0 {
+  %1 = tail call fast <4 x double> @llvm.sqrt.v4f64(<4 x double> %a) #4
+  %2 = fdiv fast <4 x double> <double 1.000000e+00, double 1.000000e+00, double 1.000000e+00, double 1.000000e+00>, %1
+  ret <4 x double> %2
+
+; FAULT-LABEL: d4rsqrt:
+; FAULT-NEXT: BB#0
+; FAULT-NEXT: fsqrt
+; FAULT-NEXT: fsqrt
+
+; CHECK-LABEL: d4rsqrt:
+; CHECK-NEXT: BB#0
+; CHECK-NEXT: fmov
+; CHECK-NEXT: frsqrte
+; CHECK: frsqrte
+}
+
 attributes #0 = { nounwind "unsafe-fp-math"="true" }