[AArch64] Add missing vrnd intrinsics

[miyengar]

Summary

This patch adds 8 missing intrinsics as specified in the Arm ACLE document section 2.12.1.1 : https://arm-software.github.io/acle/neon_intrinsics/advsimd.html#rounding-3

The intrinsics implemented are:

• vrnd32z_f64
• vrnd32zq_f64
• vrnd64z_f64
• vrnd64zq_f64
• vrnd32x_f64
• vrnd32xq_f64
• vrnd64x_f64
• vrnd64xq_f64

[dmgreen]

Sounds good. Can you make sure you upload with context, it makes the patches easier to read in phabricator: https://llvm.org/docs/DeveloperPolicy.html#making-and-submitting-a-patch.

llvm/lib/Target/AArch64/AArch64InstrFormats.td

6297 ↗

(On Diff #552719)

This looks like it is defining a new instruction. Does that already exist somewhere? Probably from somewhere like FRIntNNT.

[miyengar]

Thank you for the feedback! I've added an amended patch using the pre-existing instruction. Also, I've tried to submit the patch with context this time.

llvm/lib/Target/AArch64/AArch64InstrFormats.td

6297 ↗

(On Diff #552719)

Ah thanks! I've amended this to use the already existing instruction (FRINT32ZDr) as defined in FRIntNNT

[dmgreen]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

6309 ↗

(On Diff #553104)

I think the instructions in this multiclass are the vector variants. Can the pattern be moved to the FRIntNNT/SingleOperandFPNo16 class?

[miyengar]

Moved v1f64 -> Dr (Vector to Scalar) Pattern from SIMDTwoVectorSD to FRIntNNTVector

[miyengar]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

6309 ↗

(On Diff #553104)

Thanks for the comment. I have tried to do this, but have run into a problem:

This pattern targets the LLVM internal intrinsic: int_aarch64_neon_frint***_v1f64, as referenced in AArch64InstrInfo.td:

defm FRINT32Z : FRIntNNTVector<0, 0, "frint32z", int_aarch64_neon_frint32z>;

It pattern matches 'v1f64' to 'Dr', which corresponds to a scalar instruction defined in FRIntNNT/SingleOperandFPNo16 - I.e. it is matching the vector variant to the scalar variant.

Moving the pattern to FRIntNNT results in an error "Cannot select: intrinsic %llvm.aarch64.neon.frint64x" as FRIntNNT is designed to match the intrinsics whose name does not contain "neon". Since the current patch deals specifically with the neon variant, I don't see an uncomplicated way to modify FRIntNNT to accommodate both variants of such intrinsic (neon and vanilla).

I have moved the pattern from SIMDTwoVectorSD to FRIntNNTVector though which is hopefully a better place to put it.

Does this seem like a sensible solution? Thanks!

[dmgreen]

Other than where the pattern gets defined this looks OK to me.

llvm/lib/Target/AArch64/AArch64InstrFormats.td

6309 ↗

(On Diff #553104)

I see, Because there is a difference between int_aarch64_frint32z and int_aarch64_neon_frint32z.

Could you pass int_aarch64_neon_frint32z to FRIntNNT too, or make them free patterns in AArch64InstrInfo.td? Otherwise this is using the class to generate vector variants of FRINT32Z to also generate the scalar FRINT32ZDr instruction too. It likely doesn't matter a huge amount, but it feels like it breaks the encapsulation of the FRIntNNTVector class.

[miyengar]

Moved pattern from AArchInstrFormats.td to AArchInstrInfo.td

[miyengar]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

6309 ↗

(On Diff #553104)

Ah okay, this makes sense, thank you!

I have moved the patterns from FRIntNNTVector to make them free patterns in AArch64InstrInfo.td instead - this should preserve the encapsulation of the FRIntNNTVector class.

[dmgreen]

Thanks. LGTM

clang/include/clang/Basic/arm_neon.td

 def FRINTI_S64 : SInst<"vrndi", "..", "dQd">;
 }
 
 let ArchGuard = "defined(__aarch64__)", TargetGuard = "v8.5a" in {
 def FRINT32X_S32 : SInst<"vrnd32x", "..", "fQf">;
 def FRINT32Z_S32 : SInst<"vrnd32z", "..", "fQf">;
 def FRINT64X_S32 : SInst<"vrnd64x", "..", "fQf">;
 def FRINT64Z_S32 : SInst<"vrnd64z", "..", "fQf">;
+
+def FRINT32X_S64 : SInst<"vrnd32x", "..", "dQd">;
+def FRINT32Z_S64 : SInst<"vrnd32z", "..", "dQd">;
+def FRINT64X_S64 : SInst<"vrnd64x", "..", "dQd">;
+def FRINT64Z_S64 : SInst<"vrnd64z", "..", "dQd">;
 }
 
 ////////////////////////////////////////////////////////////////////////////////
 // MaxNum/MinNum Floating Point
 
 let ArchGuard = "__ARM_ARCH >= 8 && defined(__ARM_FEATURE_NUMERIC_MAXMIN)" in {
 def FMAXNM_S32 : SInst<"vmaxnm", "...", "fQf">;
 def FMINNM_S32 : SInst<"vminnm", "...", "fQf">;

clang/lib/CodeGen/CGBuiltin.cpp

static const ARMVectorIntrinsicInfo AArch64SIMDIntrinsicMap[] = {
   NEONMAP1(vrax1q_u64, aarch64_crypto_rax1, 0),
   NEONMAP2(vrecpe_v, aarch64_neon_frecpe, aarch64_neon_urecpe, 0),
   NEONMAP2(vrecpeq_v, aarch64_neon_frecpe, aarch64_neon_urecpe, 0),
   NEONMAP1(vrecps_v, aarch64_neon_frecps, Add1ArgType),
   NEONMAP1(vrecpsq_v, aarch64_neon_frecps, Add1ArgType),
   NEONMAP2(vrhadd_v, aarch64_neon_urhadd, aarch64_neon_srhadd, Add1ArgType | UnsignedAlts),
   NEONMAP2(vrhaddq_v, aarch64_neon_urhadd, aarch64_neon_srhadd, Add1ArgType | UnsignedAlts),
   NEONMAP1(vrnd32x_f32, aarch64_neon_frint32x, Add1ArgType),
+  NEONMAP1(vrnd32x_f64, aarch64_neon_frint32x, Add1ArgType),
   NEONMAP1(vrnd32xq_f32, aarch64_neon_frint32x, Add1ArgType),
+  NEONMAP1(vrnd32xq_f64, aarch64_neon_frint32x, Add1ArgType),
   NEONMAP1(vrnd32z_f32, aarch64_neon_frint32z, Add1ArgType),
+  NEONMAP1(vrnd32z_f64, aarch64_neon_frint32z, Add1ArgType),
   NEONMAP1(vrnd32zq_f32, aarch64_neon_frint32z, Add1ArgType),
+  NEONMAP1(vrnd32zq_f64, aarch64_neon_frint32z, Add1ArgType),
   NEONMAP1(vrnd64x_f32, aarch64_neon_frint64x, Add1ArgType),
+  NEONMAP1(vrnd64x_f64, aarch64_neon_frint64x, Add1ArgType),
   NEONMAP1(vrnd64xq_f32, aarch64_neon_frint64x, Add1ArgType),
+  NEONMAP1(vrnd64xq_f64, aarch64_neon_frint64x, Add1ArgType),
   NEONMAP1(vrnd64z_f32, aarch64_neon_frint64z, Add1ArgType),
+  NEONMAP1(vrnd64z_f64, aarch64_neon_frint64z, Add1ArgType),
   NEONMAP1(vrnd64zq_f32, aarch64_neon_frint64z, Add1ArgType),
+  NEONMAP1(vrnd64zq_f64, aarch64_neon_frint64z, Add1ArgType),
   NEONMAP0(vrndi_v),
   NEONMAP0(vrndiq_v),
   NEONMAP2(vrshl_v, aarch64_neon_urshl, aarch64_neon_srshl, Add1ArgType | UnsignedAlts),
   NEONMAP2(vrshlq_v, aarch64_neon_urshl, aarch64_neon_srshl, Add1ArgType | UnsignedAlts),
   NEONMAP2(vrshr_n_v, aarch64_neon_urshl, aarch64_neon_srshl, UnsignedAlts),
   NEONMAP2(vrshrq_n_v, aarch64_neon_urshl, aarch64_neon_srshl, UnsignedAlts),
   NEONMAP2(vrsqrte_v, aarch64_neon_frsqrte, aarch64_neon_ursqrte, 0),
   NEONMAP2(vrsqrteq_v, aarch64_neon_frsqrte, aarch64_neon_ursqrte, 0),

   case NEON::BI__builtin_neon_vrndh_f16: {
     Ops.push_back(EmitScalarExpr(E->getArg(0)));
     Int = Builder.getIsFPConstrained()
               ? Intrinsic::experimental_constrained_trunc
               : Intrinsic::trunc;
     return EmitNeonCall(CGM.getIntrinsic(Int, HalfTy), Ops, "vrndz");
   }
   case NEON::BI__builtin_neon_vrnd32x_f32:
-  case NEON::BI__builtin_neon_vrnd32xq_f32: {
+  case NEON::BI__builtin_neon_vrnd32xq_f32:
+  case NEON::BI__builtin_neon_vrnd32x_f64:
+  case NEON::BI__builtin_neon_vrnd32xq_f64: {
     Ops.push_back(EmitScalarExpr(E->getArg(0)));
     Int = Intrinsic::aarch64_neon_frint32x;
     return EmitNeonCall(CGM.getIntrinsic(Int, Ty), Ops, "vrnd32x");
   }
   case NEON::BI__builtin_neon_vrnd32z_f32:
-  case NEON::BI__builtin_neon_vrnd32zq_f32: {
+  case NEON::BI__builtin_neon_vrnd32zq_f32:
+  case NEON::BI__builtin_neon_vrnd32z_f64:
+  case NEON::BI__builtin_neon_vrnd32zq_f64: {
     Ops.push_back(EmitScalarExpr(E->getArg(0)));
     Int = Intrinsic::aarch64_neon_frint32z;
     return EmitNeonCall(CGM.getIntrinsic(Int, Ty), Ops, "vrnd32z");
   }
   case NEON::BI__builtin_neon_vrnd64x_f32:
-  case NEON::BI__builtin_neon_vrnd64xq_f32: {
+  case NEON::BI__builtin_neon_vrnd64xq_f32:
+  case NEON::BI__builtin_neon_vrnd64x_f64:
+  case NEON::BI__builtin_neon_vrnd64xq_f64: {
     Ops.push_back(EmitScalarExpr(E->getArg(0)));
     Int = Intrinsic::aarch64_neon_frint64x;
     return EmitNeonCall(CGM.getIntrinsic(Int, Ty), Ops, "vrnd64x");
   }
   case NEON::BI__builtin_neon_vrnd64z_f32:
-  case NEON::BI__builtin_neon_vrnd64zq_f32: {
+  case NEON::BI__builtin_neon_vrnd64zq_f32:
+  case NEON::BI__builtin_neon_vrnd64z_f64:
+  case NEON::BI__builtin_neon_vrnd64zq_f64: {
     Ops.push_back(EmitScalarExpr(E->getArg(0)));
     Int = Intrinsic::aarch64_neon_frint64z;
     return EmitNeonCall(CGM.getIntrinsic(Int, Ty), Ops, "vrnd64z");
   }
   case NEON::BI__builtin_neon_vrnd_v:
   case NEON::BI__builtin_neon_vrndq_v: {
     Int = Builder.getIsFPConstrained()
               ? Intrinsic::experimental_constrained_trunc

clang/test/CodeGen/aarch64-v8.5a-neon-frint3264-intrinsic.c

 }
 
 // CHECK-LABEL: test_vrnd64zq_f32
 // CHECK:  [[RND:%.*]] =  call <4 x float> @llvm.aarch64.neon.frint64z.v4f32(<4 x float> %a)
 // CHECK:  ret <4 x float> [[RND]]
 float32x4_t test_vrnd64zq_f32(float32x4_t a) {
   return vrnd64zq_f32(a);
 }
+
+// CHECK-LABEL: test_vrnd32x_f64
+// CHECK:  [[RND:%.*]] =  call <1 x double> @llvm.aarch64.neon.frint32x.v1f64(<1 x double> %a)
+// CHECK:  ret <1 x double> [[RND]]
+float64x1_t test_vrnd32x_f64(float64x1_t a) {
+  return vrnd32x_f64(a);
+}
+
+// CHECK-LABEL: test_vrnd32xq_f64
+// CHECK:  [[RND:%.*]] =  call <2 x double> @llvm.aarch64.neon.frint32x.v2f64(<2 x double> %a)
+// CHECK:  ret <2 x double> [[RND]]
+float64x2_t test_vrnd32xq_f64(float64x2_t a) {
+  return vrnd32xq_f64(a);
+}
+
+// CHECK-LABEL: test_vrnd32z_f64
+// CHECK:  [[RND:%.*]] =  call <1 x double> @llvm.aarch64.neon.frint32z.v1f64(<1 x double> %a)
+// CHECK:  ret <1 x double> [[RND]]
+float64x1_t test_vrnd32z_f64(float64x1_t a) {
+  return vrnd32z_f64(a);
+}
+
+// CHECK-LABEL: test_vrnd32zq_f64
+// CHECK:  [[RND:%.*]] =  call <2 x double> @llvm.aarch64.neon.frint32z.v2f64(<2 x double> %a)
+// CHECK:  ret <2 x double> [[RND]]
+float64x2_t test_vrnd32zq_f64(float64x2_t a) {
+  return vrnd32zq_f64(a);
+}
+
+// CHECK-LABEL: test_vrnd64x_f64
+// CHECK:  [[RND:%.*]] =  call <1 x double> @llvm.aarch64.neon.frint64x.v1f64(<1 x double> %a)
+// CHECK:  ret <1 x double> [[RND]]
+float64x1_t test_vrnd64x_f64(float64x1_t a) {
+  return vrnd64x_f64(a);
+}
+
+// CHECK-LABEL: test_vrnd64xq_f64
+// CHECK:  [[RND:%.*]] =  call <2 x double> @llvm.aarch64.neon.frint64x.v2f64(<2 x double> %a)
+// CHECK:  ret <2 x double> [[RND]]
+float64x2_t test_vrnd64xq_f64(float64x2_t a) {
+  return vrnd64xq_f64(a);
+}
+
+// CHECK-LABEL: test_vrnd64z_f64
+// CHECK:  [[RND:%.*]] =  call <1 x double> @llvm.aarch64.neon.frint64z.v1f64(<1 x double> %a)
+// CHECK:  ret <1 x double> [[RND]]
+float64x1_t test_vrnd64z_f64(float64x1_t a) {
+  return vrnd64z_f64(a);
+}
+
+// CHECK-LABEL: test_vrnd64zq_f64
+// CHECK:  [[RND:%.*]] =  call <2 x double> @llvm.aarch64.neon.frint64z.v2f64(<2 x double> %a)
+// CHECK:  ret <2 x double> [[RND]]
+float64x2_t test_vrnd64zq_f64(float64x2_t a) {
+  return vrnd64zq_f64(a);
+}

llvm/lib/Target/AArch64/AArch64InstrInfo.td

 
 let Predicates = [HasFRInt3264] in {
   defm FRINT32Z : FRIntNNT<0b00, "frint32z", int_aarch64_frint32z>;
   defm FRINT64Z : FRIntNNT<0b10, "frint64z", int_aarch64_frint64z>;
   defm FRINT32X : FRIntNNT<0b01, "frint32x", int_aarch64_frint32x>;
   defm FRINT64X : FRIntNNT<0b11, "frint64x", int_aarch64_frint64x>;
 } // HasFRInt3264
 
+// Pattern to convert 1x64 vector intrinsics to equivalent scalar instructions
+def : Pat<(v1f64 (int_aarch64_neon_frint32z (v1f64 FPR64:$Rn))),
+          (FRINT32ZDr FPR64:$Rn)>;
+def : Pat<(v1f64 (int_aarch64_neon_frint64z (v1f64 FPR64:$Rn))),
+          (FRINT64ZDr FPR64:$Rn)>;
+def : Pat<(v1f64 (int_aarch64_neon_frint32x (v1f64 FPR64:$Rn))),
+          (FRINT32XDr FPR64:$Rn)>;
+def : Pat<(v1f64 (int_aarch64_neon_frint64x (v1f64 FPR64:$Rn))),
+          (FRINT64XDr FPR64:$Rn)>;
+
 // Emitting strict_lrint as two instructions is valid as any exceptions that
 // occur will happen in exactly one of the instructions (e.g. if the input is
 // not an integer the inexact exception will happen in the FRINTX but not then
 // in the FCVTZS as the output of FRINTX is an integer).
 let Predicates = [HasFullFP16] in {
   def : Pat<(i32 (any_lrint f16:$Rn)),
             (FCVTZSUWHr (FRINTXHr f16:$Rn))>;
   def : Pat<(i64 (any_lrint f16:$Rn)),

llvm/test/CodeGen/AArch64/v8.5a-neon-frint3264-intrinsic.ll

 define dso_local <4 x float> @t_vrnd64zq_f32(<4 x float> %a) {
 ; CHECK-LABEL: t_vrnd64zq_f32:
 ; CHECK:         frint64z v0.4s, v0.4s
 ; CHECK-NEXT:    ret
 entry:
   %val = tail call <4 x float> @llvm.aarch64.neon.frint64z.v4f32(<4 x float> %a)
   ret <4 x float> %val
 }
+
+declare <1 x double> @llvm.aarch64.neon.frint32x.v1f64(<1 x double>)
+declare <2 x double> @llvm.aarch64.neon.frint32x.v2f64(<2 x double>)
+declare <1 x double> @llvm.aarch64.neon.frint32z.v1f64(<1 x double>)
+declare <2 x double> @llvm.aarch64.neon.frint32z.v2f64(<2 x double>)
+
+define dso_local <1 x double> @t_vrnd32x_f64(<1 x double> %a) {
+; CHECK-LABEL: t_vrnd32x_f64:
+; CHECK:         frint32x d0, d0
+; CHECK-NEXT:    ret
+entry:
+  %val = tail call <1 x double> @llvm.aarch64.neon.frint32x.v1f64(<1 x double> %a)
+  ret <1 x double> %val
+}
+
+define dso_local <2 x double> @t_vrnd32xq_f64(<2 x double> %a) {
+; CHECK-LABEL: t_vrnd32xq_f64:
+; CHECK:         frint32x v0.2d, v0.2d
+; CHECK-NEXT:    ret
+entry:
+  %val = tail call <2 x double> @llvm.aarch64.neon.frint32x.v2f64(<2 x double> %a)
+  ret <2 x double> %val
+}
+
+define dso_local <1 x double> @t_vrnd32z_f64(<1 x double> %a) {
+; CHECK-LABEL: t_vrnd32z_f64:
+; CHECK:         frint32z d0, d0
+; CHECK-NEXT:    ret
+entry:
+  %val = tail call <1 x double> @llvm.aarch64.neon.frint32z.v1f64(<1 x double> %a)
+  ret <1 x double> %val
+}
+
+define dso_local <2 x double> @t_vrnd32zq_f64(<2 x double> %a) {
+; CHECK-LABEL: t_vrnd32zq_f64:
+; CHECK:         frint32z v0.2d, v0.2d
+; CHECK-NEXT:    ret
+entry:
+  %val = tail call <2 x double> @llvm.aarch64.neon.frint32z.v2f64(<2 x double> %a)
+  ret <2 x double> %val
+}
+
+declare <1 x double> @llvm.aarch64.neon.frint64x.v1f64(<1 x double>)
+declare <2 x double> @llvm.aarch64.neon.frint64x.v2f64(<2 x double>)
+declare <1 x double> @llvm.aarch64.neon.frint64z.v1f64(<1 x double>)
+declare <2 x double> @llvm.aarch64.neon.frint64z.v2f64(<2 x double>)
+
+define dso_local <1 x double> @t_vrnd64x_f64(<1 x double> %a) {
+; CHECK-LABEL: t_vrnd64x_f64:
+; CHECK:         frint64x d0, d0
+; CHECK-NEXT:    ret
+entry:
+  %val = tail call <1 x double> @llvm.aarch64.neon.frint64x.v1f64(<1 x double> %a)
+  ret <1 x double> %val
+}
+
+define dso_local <2 x double> @t_vrnd64xq_f64(<2 x double> %a) {
+; CHECK-LABEL: t_vrnd64xq_f64:
+; CHECK:         frint64x v0.2d, v0.2d
+; CHECK-NEXT:    ret
+entry:
+  %val = tail call <2 x double> @llvm.aarch64.neon.frint64x.v2f64(<2 x double> %a)
+  ret <2 x double> %val
+}
+
+define dso_local <1 x double> @t_vrnd64z_f64(<1 x double> %a) {
+; CHECK-LABEL: t_vrnd64z_f64:
+; CHECK:         frint64z d0, d0
+; CHECK-NEXT:    ret
+entry:
+  %val = tail call <1 x double> @llvm.aarch64.neon.frint64z.v1f64(<1 x double> %a)
+  ret <1 x double> %val
+}
+
+define dso_local <2 x double> @t_vrnd64zq_f64(<2 x double> %a) {
+; CHECK-LABEL: t_vrnd64zq_f64:
+; CHECK:         frint64z v0.2d, v0.2d
+; CHECK-NEXT:    ret
+entry:
+  %val = tail call <2 x double> @llvm.aarch64.neon.frint64z.v2f64(<2 x double> %a)
+  ret <2 x double> %val
+}