[ARM] Support for v4f16 and v8f16 vectors

[SjoerdMeijer]

Summary

This is the groundwork for the Armv8.2-A FP16
vector intrinsics, which uses v4f16 and v8f16 vector operands
and return values. All the moving parts are tested with two
intrinsics, a 1-operand v8f16 and a 2-operand v4f16 intrinsic. In a
follow-up patch the rest of the intrinsics and tests will be added.

[t.p.northover]

It looks like there's nothing testing the bitconvert patterns here.

Also, as a general question, didn't the Clang changes go in (and get reverted) before this? That seems pretty dodgy to me; I'm not entirely happy about having user-visible but broken intrinsics in arm_acle.h.

lib/Target/ARM/ARMInstrNEON.td

7020

I know big-endian is a pain in the neck that hardly anyone actually uses, but we probably shouldn't just let it bit-rot

[SjoerdMeijer]

Hi Tim,
Thanks for your comments. I will try to answer your general question first (if I understand that correctly).
There should be no dodgy business going on here. I am trying to avoid exactly that, and I've reverted the
Clang and user visible part when that showed problems in testing. Please note that this reverted the
A32 intrinsics, the A64 are still in and should be okay. The FP16 A32 intrinsics were behaving as expected,
but there was some interaction with existing intrinsics and __fp16. The reason for that is that the author
of the intrinsics patches changed passing vectors of i16 types, to vector of f16s.
With the user-visible part reverted, I tried fixing the backend first with this patch. But I've just noticed that
this patch doesn't solve that case, which is obviously what I also tried to achieve here. There are some issues now
with legalising f16 vectors (when fullfp16 is not enabled). I am now first going to rethink why we want to pass
f16 vectors instead of sticking to i16s..
Cheers.

[t.p.northover]

Ah good, thanks for the explanation. Sounds like you were already doing exactly what I was thinking of and I started paying attention half-way through.

I am now first going to rethink why we want to pass f16 vectors instead of sticking to i16.

I think either could work. Longer term we probably want <N x half> for tidiness, but It's probably not essential to begin with.

[SjoerdMeijer]

I have uploaded (companion) Clang patch: https://reviews.llvm.org/D44561
this passes the Half Type when it is appropriate to do so, and i16 otherwise.
Thus, the normal neon intrinsics work as before and as expected, and with this
LLVM patch for f16 vectors, the FP16 vector intrinsics are fine too.

About the bitconverts, they were actually necessary to get the code generation
working for the 2 intrinsics in the regression tests. They run in hard and
softfp mode, and these bitconverts patterns were necessary for the softfp case,
but I will double check to be sure.

If we are happy with this, then I will first add (or actually reenable) the AArch32
FP16 vector intrinsics to clang again, and then complete the rest of the LLVM
codegen tests.

[t.p.northover]

Yep, I thought the bitconverts were a good idea, but should also have the variants that trigger for IsBE.

[SjoerdMeijer]

Yep, I thought the bitconverts were a good idea, but should also have the variants that trigger for IsBE.

Ah yes, sorry, I misunderstood and missed that. Will fix that.

[SjoerdMeijer]

Added big-endian bitconvert patterns and test.

[t.p.northover]

Thanks. This looks fine now too.

lib/Target/ARM/ARMCallingConv.h

case MVT::i32: {
     // First consume all registers that would give an unaligned object. Whether
     // we go on stack or in regs, no-one will be using them in future.
     unsigned RegAlign = alignTo(Align, 4) / 4;
     while (RegIdx % RegAlign != 0 && RegIdx < RegList.size())
       State.AllocateReg(RegList[RegIdx++]);
 
     break;
   }
+  case MVT::f16:
   case MVT::f32:
     RegList = SRegList;
     break;
+  case MVT::v4f16:
   case MVT::f64:
     RegList = DRegList;
     break;
+  case MVT::v8f16:
   case MVT::v2f64:
     RegList = QRegList;
     break;
   default:
     llvm_unreachable("Unexpected member type for block aggregate");
     break;
   }
 

lib/Target/ARM/ARMCallingConv.td

 def CC_ARM_AAPCS : CallingConv<[
   // Handles byval parameters.
   CCIfByVal<CCPassByVal<4, 4>>,
 
   // The 'nest' parameter, if any, is passed in R12.
   CCIfNest<CCAssignToReg<[R12]>>,
 
   // Handle all vector types as either f64 or v2f64.
-  CCIfType<[v1i64, v2i32, v4i16, v8i8, v2f32], CCBitConvertToType<f64>>,
-  CCIfType<[v2i64, v4i32, v8i16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
+  CCIfType<[v1i64, v2i32, v4i16, v4f16, v8i8, v2f32], CCBitConvertToType<f64>>,
+  CCIfType<[v2i64, v4i32, v8i16, v8f16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
   // Pass SwiftSelf in a callee saved register.
   CCIfSwiftSelf<CCIfType<[i32], CCAssignToReg<[R10]>>>,
 
   // A SwiftError is passed in R8.
   CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R8]>>>,
 
   CCIfType<[f64, v2f64], CCCustom<"CC_ARM_AAPCS_Custom_f64">>,
   CCIfType<[f32], CCBitConvertToType<i32>>,
   CCDelegateTo<CC_ARM_AAPCS_Common>
 ]>;
 
 def RetCC_ARM_AAPCS : CallingConv<[
   // Handle all vector types as either f64 or v2f64.
-  CCIfType<[v1i64, v2i32, v4i16, v8i8, v2f32], CCBitConvertToType<f64>>,
-  CCIfType<[v2i64, v4i32, v8i16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
+  CCIfType<[v1i64, v2i32, v4i16, v4f16, v8i8, v2f32], CCBitConvertToType<f64>>,
+  CCIfType<[v2i64, v4i32, v8i16, v8f16,v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
   // Pass SwiftSelf in a callee saved register.
   CCIfSwiftSelf<CCIfType<[i32], CCAssignToReg<[R10]>>>,
 
   // A SwiftError is returned in R8.
   CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R8]>>>,
 
   CCIfType<[f64, v2f64], CCCustom<"RetCC_ARM_AAPCS_Custom_f64">>,
   CCIfType<[f32], CCBitConvertToType<i32>>,
 
   CCDelegateTo<RetCC_ARM_AAPCS_Common>
 ]>;
 
 //===----------------------------------------------------------------------===//
 // ARM AAPCS-VFP (EABI) Calling Convention
 // Also used for FastCC (when VFP2 or later is available)
 //===----------------------------------------------------------------------===//
 
 def CC_ARM_AAPCS_VFP : CallingConv<[
   // Handles byval parameters.
   CCIfByVal<CCPassByVal<4, 4>>,
 
   // Handle all vector types as either f64 or v2f64.
-  CCIfType<[v1i64, v2i32, v4i16, v8i8, v2f32], CCBitConvertToType<f64>>,
-  CCIfType<[v2i64, v4i32, v8i16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
+  CCIfType<[v1i64, v2i32, v4i16, v4f16, v8i8, v2f32], CCBitConvertToType<f64>>,
+  CCIfType<[v2i64, v4i32, v8i16, v8f16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
   // Pass SwiftSelf in a callee saved register.
   CCIfSwiftSelf<CCIfType<[i32], CCAssignToReg<[R10]>>>,
 
   // A SwiftError is passed in R8.
   CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R8]>>>,
 
   // HFAs are passed in a contiguous block of registers, or on the stack
   CCIfConsecutiveRegs<CCCustom<"CC_ARM_AAPCS_Custom_Aggregate">>,
 
   CCIfType<[v2f64], CCAssignToReg<[Q0, Q1, Q2, Q3]>>,
   CCIfType<[f64], CCAssignToReg<[D0, D1, D2, D3, D4, D5, D6, D7]>>,
   CCIfType<[f32], CCAssignToReg<[S0, S1, S2, S3, S4, S5, S6, S7, S8,
                                  S9, S10, S11, S12, S13, S14, S15]>>,
   CCDelegateTo<CC_ARM_AAPCS_Common>
 ]>;
 
 def RetCC_ARM_AAPCS_VFP : CallingConv<[
   // Handle all vector types as either f64 or v2f64.
-  CCIfType<[v1i64, v2i32, v4i16, v8i8, v2f32], CCBitConvertToType<f64>>,
-  CCIfType<[v2i64, v4i32, v8i16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
+  CCIfType<[v1i64, v2i32, v4i16, v4f16, v8i8, v2f32], CCBitConvertToType<f64>>,
+  CCIfType<[v2i64, v4i32, v8i16, v8f16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
   // Pass SwiftSelf in a callee saved register.
   CCIfSwiftSelf<CCIfType<[i32], CCAssignToReg<[R10]>>>,
 
   // A SwiftError is returned in R8.
   CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R8]>>>,
 
   CCIfType<[v2f64], CCAssignToReg<[Q0, Q1, Q2, Q3]>>,

lib/Target/ARM/ARMISelDAGToDAG.cpp

void ARMDAGToDAGISel::SelectVST(SDNode *N, bool isUpdating, unsigned NumVecs,
   bool is64BitVector = VT.is64BitVector();
   Align = GetVLDSTAlign(Align, dl, NumVecs, is64BitVector);
 
   unsigned OpcodeIndex;
   switch (VT.getSimpleVT().SimpleTy) {
   default: llvm_unreachable("unhandled vst type");
     // Double-register operations:
   case MVT::v8i8:  OpcodeIndex = 0; break;
+  case MVT::v4f16:
   case MVT::v4i16: OpcodeIndex = 1; break;
   case MVT::v2f32:
   case MVT::v2i32: OpcodeIndex = 2; break;
   case MVT::v1i64: OpcodeIndex = 3; break;
     // Quad-register operations:
   case MVT::v16i8: OpcodeIndex = 0; break;
+  case MVT::v8f16:
   case MVT::v8i16: OpcodeIndex = 1; break;
   case MVT::v4f32:
   case MVT::v4i32: OpcodeIndex = 2; break;
   case MVT::v2f64:
   case MVT::v2i64: OpcodeIndex = 3;
     assert(NumVecs == 1 && "v2i64 type only supported for VST1");
     break;
   }

lib/Target/ARM/ARMISelLowering.cpp

if (Subtarget->hasNEON()) {
 
     addQRTypeForNEON(MVT::v4f32);
     addQRTypeForNEON(MVT::v2f64);
     addQRTypeForNEON(MVT::v16i8);
     addQRTypeForNEON(MVT::v8i16);
     addQRTypeForNEON(MVT::v4i32);
     addQRTypeForNEON(MVT::v2i64);
 
+    if (Subtarget->hasFullFP16()) {
+      addQRTypeForNEON(MVT::v8f16);
+      addDRTypeForNEON(MVT::v4f16);
+    }
+
     // v2f64 is legal so that QR subregs can be extracted as f64 elements, but
     // neither Neon nor VFP support any arithmetic operations on it.
     // The same with v4f32. But keep in mind that vadd, vsub, vmul are natively
     // supported for v4f32.
     setOperationAction(ISD::FADD, MVT::v2f64, Expand);
     setOperationAction(ISD::FSUB, MVT::v2f64, Expand);
     setOperationAction(ISD::FMUL, MVT::v2f64, Expand);
     // FIXME: Code duplication: FDIV and FREM are expanded always, see
if (VA.isRegLoc()) {
       } else {
         const TargetRegisterClass *RC;
 
 
         if (RegVT == MVT::f16)
           RC = &ARM::HPRRegClass;
         else if (RegVT == MVT::f32)
           RC = &ARM::SPRRegClass;
-        else if (RegVT == MVT::f64)
+        else if (RegVT == MVT::f64 || RegVT == MVT::v4f16)
           RC = &ARM::DPRRegClass;
-        else if (RegVT == MVT::v2f64)
+        else if (RegVT == MVT::v2f64 || RegVT == MVT::v8f16)
           RC = &ARM::QPRRegClass;
         else if (RegVT == MVT::i32)
           RC = AFI->isThumb1OnlyFunction() ? &ARM::tGPRRegClass
                                            : &ARM::GPRRegClass;
         else
           llvm_unreachable("RegVT not supported by FORMAL_ARGUMENTS Lowering");
 
         // Transform the arguments in physical registers into virtual ones.

lib/Target/ARM/ARMInstrNEON.td

let Predicates = [IsLE] in {
   def : Pat<(v8i8  (bitconvert (v4i16 DPR:$src))), (v8i8  DPR:$src)>;
   def : Pat<(v8i8  (bitconvert (f64   DPR:$src))), (v8i8  DPR:$src)>;
   def : Pat<(v8i8  (bitconvert (v2f32 DPR:$src))), (v8i8  DPR:$src)>;
 }
 def : Pat<(f64   (bitconvert (v1i64 DPR:$src))), (f64   DPR:$src)>;
 let Predicates = [IsLE] in {
   def : Pat<(f64   (bitconvert (v2i32 DPR:$src))), (f64   DPR:$src)>;
   def : Pat<(f64   (bitconvert (v4i16 DPR:$src))), (f64   DPR:$src)>;
+  def : Pat<(f64   (bitconvert (v4f16 DPR:$src))), (f64   DPR:$src)>;
   def : Pat<(f64   (bitconvert (v8i8  DPR:$src))), (f64   DPR:$src)>;
   def : Pat<(f64   (bitconvert (v2f32 DPR:$src))), (f64   DPR:$src)>;
   def : Pat<(v2f32 (bitconvert (f64   DPR:$src))), (v2f32 DPR:$src)>;
+  def : Pat<(v4f16 (bitconvert (f64   DPR:$src))), (v4f16 DPR:$src)>;
   def : Pat<(v2f32 (bitconvert (v1i64 DPR:$src))), (v2f32 DPR:$src)>;
 }
 def : Pat<(v2f32 (bitconvert (v2i32 DPR:$src))), (v2f32 DPR:$src)>;
 let Predicates = [IsLE] in {
   def : Pat<(v2f32 (bitconvert (v4i16 DPR:$src))), (v2f32 DPR:$src)>;
   def : Pat<(v2f32 (bitconvert (v8i8  DPR:$src))), (v2f32 DPR:$src)>;
 }
 

 }
 def : Pat<(v4i32 (bitconvert (v4f32 QPR:$src))), (v4i32 QPR:$src)>;
 let Predicates = [IsLE] in {
   def : Pat<(v8i16 (bitconvert (v2i64 QPR:$src))), (v8i16 QPR:$src)>;
   def : Pat<(v8i16 (bitconvert (v4i32 QPR:$src))), (v8i16 QPR:$src)>;
   def : Pat<(v8i16 (bitconvert (v16i8 QPR:$src))), (v8i16 QPR:$src)>;
   def : Pat<(v8i16 (bitconvert (v2f64 QPR:$src))), (v8i16 QPR:$src)>;
   def : Pat<(v8i16 (bitconvert (v4f32 QPR:$src))), (v8i16 QPR:$src)>;
+  def : Pat<(v8f16 (bitconvert (v2f64 QPR:$src))), (v8f16 QPR:$src)>;
   def : Pat<(v16i8 (bitconvert (v2i64 QPR:$src))), (v16i8 QPR:$src)>;
   def : Pat<(v16i8 (bitconvert (v4i32 QPR:$src))), (v16i8 QPR:$src)>;
   def : Pat<(v16i8 (bitconvert (v8i16 QPR:$src))), (v16i8 QPR:$src)>;
   def : Pat<(v16i8 (bitconvert (v2f64 QPR:$src))), (v16i8 QPR:$src)>;
   def : Pat<(v16i8 (bitconvert (v4f32 QPR:$src))), (v16i8 QPR:$src)>;
   def : Pat<(v4f32 (bitconvert (v2i64 QPR:$src))), (v4f32 QPR:$src)>;
 }
 def : Pat<(v4f32 (bitconvert (v4i32 QPR:$src))), (v4f32 QPR:$src)>;
 let Predicates = [IsLE] in {
   def : Pat<(v4f32 (bitconvert (v8i16 QPR:$src))), (v4f32 QPR:$src)>;
   def : Pat<(v4f32 (bitconvert (v16i8 QPR:$src))), (v4f32 QPR:$src)>;
   def : Pat<(v4f32 (bitconvert (v2f64 QPR:$src))), (v4f32 QPR:$src)>;
 }
 def : Pat<(v2f64 (bitconvert (v2i64 QPR:$src))), (v2f64 QPR:$src)>;
 let Predicates = [IsLE] in {
   def : Pat<(v2f64 (bitconvert (v4i32 QPR:$src))), (v2f64 QPR:$src)>;
   def : Pat<(v2f64 (bitconvert (v8i16 QPR:$src))), (v2f64 QPR:$src)>;
+  def : Pat<(v2f64 (bitconvert (v8f16 QPR:$src))), (v2f64 QPR:$src)>;
   def : Pat<(v2f64 (bitconvert (v16i8 QPR:$src))), (v2f64 QPR:$src)>;
   def : Pat<(v2f64 (bitconvert (v4f32 QPR:$src))), (v2f64 QPR:$src)>;
 }
 
 let Predicates = [IsBE] in {
   // 64 bit conversions
   def : Pat<(v1i64 (bitconvert (v2i32 DPR:$src))), (VREV64d32 DPR:$src)>;
   def : Pat<(v1i64 (bitconvert (v4i16 DPR:$src))), (VREV64d16 DPR:$src)>;

test/CodeGen/ARM/fp16-intrinsic-vector-1op.ll

+; RUN: llc < %s -mtriple=arm-none-eabi -mattr=+v8.2a,+fullfp16,+neon  -float-abi=hard   | FileCheck %s --check-prefixes=CHECK,CHECK-HARD
+; RUN: llc < %s -mtriple=arm-none-eabi -mattr=+v8.2a,+fullfp16,+neon  | FileCheck %s --check-prefixes=CHECK,CHECK-SOFTFP
+
+declare <8 x half> @llvm.fabs.v8f16(<8 x half>)
+
+define dso_local <8 x half> @t_vabsq_f16(<8 x half> %a) {
+; CHECK-LABEL:      t_vabsq_f16:
+
+; CHECK-HARD:         vabs.f16  q0, q0
+; CHECK-HARD-NEXT:    bx  lr
+
+; CHECK-SOFTFP:       vmov  d{{.*}}, r2, r3
+; CHECK-SOFTFP:       vmov  d{{.*}}, r0, r1
+; CHECK-SOFTFP:       vabs.f16  q{{.*}}, q{{.*}}
+; CHECK-SOFTFP:       vmov  r0, r1, d{{.*}}
+; CHECK-SOFTFP:       vmov  r2, r3, d{{.*}}
+; CHECK-SOFTFP:       bx  lr
+
+entry:
+  %vabs1.i = tail call <8 x half> @llvm.fabs.v8f16(<8 x half> %a) #3
+  ret <8 x half> %vabs1.i
+}
+

test/CodeGen/ARM/fp16-intrinsic-vector-2op.ll

+; RUN: llc < %s -mtriple=arm-none-eabi -mattr=+v8.2a,+fullfp16,+neon  -float-abi=hard   | FileCheck %s --check-prefixes=CHECK,CHECK-HARD
+; RUN: llc < %s -mtriple=arm-none-eabi -mattr=+v8.2a,+fullfp16,+neon  | FileCheck %s --check-prefixes=CHECK,CHECK-SOFTFP
+
+declare <4 x half> @llvm.arm.neon.vpadd.v4f16(<4 x half>, <4 x half>)
+
+define dso_local <4 x half> @t_vpadd_f16(<4 x half> %a, <4 x half> %b) {
+; CHECK:   t_vpadd_f16:
+
+; CHECK-HARD:       vpadd.f16 d0, d0, d1
+; CHECK-HARD-NEXT:  bx  lr
+
+; CHECK-SOFTFP:     vmov  [[D1:d[0-9]+]], r2, r3
+; CHECK-SOFTFP:     vmov  [[D2:d[0-9]+]], r0, r1
+; CHECK-SOFTFP:     vpadd.f16 [[D3:d[0-9]+]], [[D2]], [[D1]]
+; CHECK-SOFTFP:     vmov  r0, r1, [[D3]]
+; CHECK-SOFTFP:     bx  lr
+
+entry:
+  %vpadd_v2.i = tail call <4 x half> @llvm.arm.neon.vpadd.v4f16(<4 x half> %a, <4 x half> %b)
+  ret <4 x half> %vpadd_v2.i
+}