[AArch64] Match pairwise add/fadd pattern

[sanwou01]

Summary

D75689 turns the faddp pattern into a shuffle with vector add.

Match this new pattern in target-specific DAG combine, rather than ISel,
because legalization (for v2f32) turns it into a bit of a mess.

• extended to cover f16, f32, f64 and i64

[spatel]

We had similar codegen matching problems for x86's horizontal ops and fixed them with DAGCombiner/ISel pattern matching.
If you don't do that, then I think you're still going to miss faddp opportunities if the source/IR is already in the form with a shuffle.
Example:

typedef float float2 __attribute__((ext_vector_type(2)));

float faddp(float2 x) {
  return (__builtin_shufflevector(x, x, 1, 1) + x)[0];
}

$ clang -O1 faddp.c -S -o - -target aarch64 -mllvm -disable-vector-combine 
faddp:                                  // @faddp
	dup	v1.2s, v0.s[1]
	fadd	v0.2s, v1.2s, v0.2s
	ret

[sanwou01]

Thanks @spatel . You're right that we miss that pattern, but, so does x86 currently it seems (I don't read x86 very well so I might be wrong). Using your faddp example:

$  ./bin/clang -O1 ~/tmp/faddp.c -S -o - -target x86_64 -mllvm -disable-vector-combine
...
faddp:                                  # @faddp
        .cfi_startproc
# %bb.0:                                # %entry
        pushq   %rbp
        .cfi_def_cfa_offset 16
        .cfi_offset %rbp, -16
        movq    %rsp, %rbp
        .cfi_def_cfa_register %rbp
        movaps  %xmm0, %xmm1
        shufps  $85, %xmm0, %xmm1               # xmm1 = xmm1[1,1],xmm0[1,1]
        addss   %xmm0, %xmm1
        movaps  %xmm1, %xmm0
        popq    %rbp
        .cfi_def_cfa %rsp, 8
        retq

I did find scalarizeBinOpOfSplats in DAGCombiner but that doesn't seem to work here, nor do any of the other patterns in SimplifyVBinOp.

That said, it does seem to make sense to do this in DAGCombiner, thanks for the suggestion. I'll try that.

[spatel]

In D87231#2260558, @sanwou01 wrote:

Thanks @spatel . You're right that we miss that pattern, but, so does x86 currently it seems (I don't read x86 very well so I might be wrong).

Horizontal math ops are a special case for x86 (not all targets support them and even fewer prefer them for performance), so we need to make a CPU subtarget adjustment to see if that example is working:

$ clang -O1 faddp.c -S -o - -target x86_64 -mllvm -disable-vector-combine -march=btver2
  vhaddps	%xmm0, %xmm0, %xmm0

I did find scalarizeBinOpOfSplats in DAGCombiner but that doesn't seem to work here, nor do any of the other patterns in SimplifyVBinOp.

The x86 horizontal transforms are specialized because the HW instructions themselves are weird - no sane target would ever create that functionality from scratch. :)
See "LowerToHorizontalOp" and "lowerAddSubToHorizontalOp" in X86ISelLowering.cpp.

That said, there may still be room to improve the cost models and/or usage here, but I'm not sure exactly how to adjust it. For example, we might match this pattern as a 2-way pairwise reduction?

[RKSimon]

I think the best option would be to start generating reduction intrinsics in IR, ensure cost models are accurate for them and do all we can to coax the vectorizers to recognize them (inc. partial reduction patterns) - I've been playing whack-a-mole with improving HorizOp patterns in the backend for years now and its not fun any more :-(

The plan to drop the experimental tags from the reduction intrinsics keeps getting delayed - I think due to a couple of minor issues - efficient non pow-2 type handling and inf/nan handling for fp types are the ones that @spatel reminded me about recently.

[RKSimon]

llvm/test/CodeGen/AArch64/combine-vectors-faddp.ll

2 ↗

(On Diff #290249)

VectorCombine tests should be put in llvm\test\Transforms\VectorCombine

[sanwou01]

Thanks for the feedback. I agree that ideally we'd be generating reduction intrinsics in IR and matching that in the backends. I don't think the pairwise add can be represented with the current intrinsics though: we'd need a <2 x float> variant, or a predicated version of the <4 x float> intrinsic to do this for strict FP math, I believe.

So at least for the moment I'll continue playing whack-a-mole and match the pattern in AArch64 ISel lowering.

[sanwou01]

Rework to match faddp in AArch64 ISel lowering

[dmgreen]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11598

Could this apply equally for f16/f64 as well?

[sanwou01]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11598

I think so. Looks like we're missing the f16 FADDP pattern in ISel so might as well add that too.

Similar for i64 ADDP actually.

[sanwou01]

Extend to f16, f32, f64 and i64

[dmgreen]

Thanks for making the extra fp16 patterns too. LGTM

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11627

SDLoc(N) -> DL

[sanwou01]

Fix for when there is no fp16 faddp + testing

[sanwou01]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

11627

Huh, I missed those, thanks! This'll be fixed when I land this change.

[sanwou01]

Committed as d5fd3d9b903e

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

AArch64TargetLowering::AArch64TargetLowering(const TargetMachine &TM,
   setTargetDAGCombine(ISD::MUL);
 
   setTargetDAGCombine(ISD::SELECT);
   setTargetDAGCombine(ISD::VSELECT);
 
   setTargetDAGCombine(ISD::INTRINSIC_VOID);
   setTargetDAGCombine(ISD::INTRINSIC_W_CHAIN);
   setTargetDAGCombine(ISD::INSERT_VECTOR_ELT);
+  setTargetDAGCombine(ISD::EXTRACT_VECTOR_ELT);
 
   setTargetDAGCombine(ISD::GlobalAddress);
 
   // In case of strict alignment, avoid an excessive number of byte wide stores.
   MaxStoresPerMemsetOptSize = 8;
   MaxStoresPerMemset = Subtarget->requiresStrictAlign()
                        ? MaxStoresPerMemsetOptSize : 32;
 
performVectorTruncateCombine(SDNode *N, TargetLowering::DAGCombinerInfo &DCI,
   unsigned HADDOpc = IsSignExtend
                          ? (IsRHADD ? AArch64ISD::SRHADD : AArch64ISD::SHADD)
                          : (IsRHADD ? AArch64ISD::URHADD : AArch64ISD::UHADD);
   SDValue ResultHADD = DAG.getNode(HADDOpc, DL, VT, OpA, OpB);
 
   return ResultHADD;
 }
 
+static bool hasPairwiseAdd(unsigned Opcode, EVT VT, bool FullFP16) {
+  switch (Opcode) {
+  case ISD::FADD:
+    return (FullFP16 && VT == MVT::f16) || VT == MVT::f32 || VT == MVT::f64;
+  case ISD::ADD:
+    return VT == MVT::i64;
+  default:
+    return false;
+  }
+}
+
+static SDValue performExtractVectorEltCombine(SDNode *N, SelectionDAG &DAG) {
+  SDValue N0 = N->getOperand(0), N1 = N->getOperand(1);
+  ConstantSDNode *ConstantN1 = dyn_cast<ConstantSDNode>(N1);
+
+  EVT VT = N->getValueType(0);
+  const bool FullFP16 =
+      static_cast<const AArch64Subtarget &>(DAG.getSubtarget()).hasFullFP16();
+
+  // Rewrite for pairwise fadd pattern
+  //   (f32 (extract_vector_elt
+  //           (fadd (vXf32 Other)
+  //                 (vector_shuffle (vXf32 Other) undef <1,X,...> )) 0))
+  // ->
+  //   (f32 (fadd (extract_vector_elt (vXf32 Other) 0)
+  //              (extract_vector_elt (vXf32 Other) 1))
+  if (ConstantN1 && ConstantN1->getZExtValue() == 0 &&
+      hasPairwiseAdd(N0->getOpcode(), VT, FullFP16)) {
+    SDLoc DL(N0);
+    SDValue N00 = N0->getOperand(0);
+    SDValue N01 = N0->getOperand(1);
+
+    ShuffleVectorSDNode *Shuffle = dyn_cast<ShuffleVectorSDNode>(N01);
+    SDValue Other = N00;
+
+    // And handle the commutative case.
+    if (!Shuffle) {
+      Shuffle = dyn_cast<ShuffleVectorSDNode>(N00);
+      Other = N01;
+    }
+
+    if (Shuffle && Shuffle->getMaskElt(0) == 1 &&
+        Other == Shuffle->getOperand(0)) {
+      return DAG.getNode(N0->getOpcode(), DL, VT,
+                         DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, VT, Other,
+                                     DAG.getConstant(0, SDLoc(N), MVT::i64)),
+                         DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, VT, Other,
+                                     DAG.getConstant(1, SDLoc(N), MVT::i64)));
+    }
+  }
+
+  return SDValue();
+}
+
 static SDValue performConcatVectorsCombine(SDNode *N,
                                            TargetLowering::DAGCombinerInfo &DCI,
                                            SelectionDAG &DAG) {
   SDLoc dl(N);
   EVT VT = N->getValueType(0);
   SDValue N0 = N->getOperand(0), N1 = N->getOperand(1);
   unsigned N0Opc = N0->getOpcode(), N1Opc = N1->getOpcode();
 
SDValue AArch64TargetLowering::PerformDAGCombine(SDNode *N,
   case AArch64ISD::DUP:
     return performPostLD1Combine(N, DCI, false);
   case AArch64ISD::NVCAST:
     return performNVCASTCombine(N);
   case AArch64ISD::UZP1:
     return performUzpCombine(N, DAG);
   case ISD::INSERT_VECTOR_ELT:
     return performPostLD1Combine(N, DCI, true);
+  case ISD::EXTRACT_VECTOR_ELT:
+    return performExtractVectorEltCombine(N, DAG);
   case ISD::INTRINSIC_VOID:
   case ISD::INTRINSIC_W_CHAIN:
     switch (cast<ConstantSDNode>(N->getOperand(1))->getZExtValue()) {
     case Intrinsic::aarch64_sve_prfb_gather_scalar_offset:
       return combineSVEPrefetchVecBaseImmOff(N, DAG, 1 /*=ScalarSizeInBytes*/);
     case Intrinsic::aarch64_sve_prfh_gather_scalar_offset:
       return combineSVEPrefetchVecBaseImmOff(N, DAG, 2 /*=ScalarSizeInBytes*/);
     case Intrinsic::aarch64_sve_prfw_gather_scalar_offset:

llvm/lib/Target/AArch64/AArch64InstrInfo.td

def : Pat<(f64 (fadd (vector_extract (v2f64 FPR128:$Rn), (i64 0)),
                      (vector_extract (v2f64 FPR128:$Rn), (i64 1)))),
            (f64 (FADDPv2i64p (v2f64 FPR128:$Rn)))>;
     // vector_extract on 64-bit vectors gets promoted to a 128 bit vector,
     // so we match on v4f32 here, not v2f32. This will also catch adding
     // the low two lanes of a true v4f32 vector.
 def : Pat<(fadd (vector_extract (v4f32 FPR128:$Rn), (i64 0)),
                 (vector_extract (v4f32 FPR128:$Rn), (i64 1))),
           (f32 (FADDPv2i32p (EXTRACT_SUBREG FPR128:$Rn, dsub)))>;
+def : Pat<(fadd (vector_extract (v8f16 FPR128:$Rn), (i64 0)),
+                (vector_extract (v8f16 FPR128:$Rn), (i64 1))),
+          (f16 (FADDPv2i16p (EXTRACT_SUBREG FPR128:$Rn, dsub)))>;
 
 // Scalar 64-bit shifts in FPR64 registers.
 def : Pat<(i64 (int_aarch64_neon_sshl (i64 FPR64:$Rn), (i64 FPR64:$Rm))),
           (SSHLv1i64 FPR64:$Rn, FPR64:$Rm)>;
 def : Pat<(i64 (int_aarch64_neon_ushl (i64 FPR64:$Rn), (i64 FPR64:$Rm))),
           (USHLv1i64 FPR64:$Rn, FPR64:$Rm)>;
 def : Pat<(i64 (int_aarch64_neon_srshl (i64 FPR64:$Rn), (i64 FPR64:$Rm))),
           (SRSHLv1i64 FPR64:$Rn, FPR64:$Rm)>;

llvm/test/CodeGen/AArch64/faddp-half.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc --mtriple=aarch64 -mattr=+fullfp16 < %s | FileCheck %s
 ; RUN: llc --mtriple=aarch64 < %s | FileCheck %s --check-prefix=CHECKNOFP16
 
 define half @faddp_2xhalf(<2 x half> %a) {
 ; CHECK-LABEL: faddp_2xhalf:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    dup v1.4h, v0.h[1]
-; CHECK-NEXT:    fadd v0.4h, v0.4h, v1.4h
-; CHECK-NEXT:    // kill: def $h0 killed $h0 killed $q0
+; CHECK-NEXT:    faddp h0, v0.2h
 ; CHECK-NEXT:    ret
 ;
 ; CHECKNOFP16-LABEL: faddp_2xhalf:
 ; CHECKNOFP16:       // %bb.0: // %entry
 ; CHECKNOFP16-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECKNOFP16-NEXT:    dup v1.4h, v0.h[1]
 ; CHECKNOFP16-NEXT:    fcvtl v1.4s, v1.4h
 ; CHECKNOFP16-NEXT:    fcvtl v0.4s, v0.4h
 ; CHECKNOFP16-NEXT:    fadd v0.4s, v0.4s, v1.4s
 ; CHECKNOFP16-NEXT:    fcvtn v0.4h, v0.4s
 ; CHECKNOFP16-NEXT:    // kill: def $h0 killed $h0 killed $q0
 ; CHECKNOFP16-NEXT:    ret
 entry:
   %shift = shufflevector <2 x half> %a, <2 x half> undef, <2 x i32> <i32 1, i32 undef>
   %0 = fadd <2 x half> %a, %shift
   %1 = extractelement <2 x half> %0, i32 0
   ret half %1
 }
 
 define half @faddp_2xhalf_commute(<2 x half> %a) {
 ; CHECK-LABEL: faddp_2xhalf_commute:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    dup v1.4h, v0.h[1]
-; CHECK-NEXT:    fadd v0.4h, v1.4h, v0.4h
-; CHECK-NEXT:    // kill: def $h0 killed $h0 killed $q0
+; CHECK-NEXT:    faddp h0, v0.2h
 ; CHECK-NEXT:    ret
 ;
 ; CHECKNOFP16-LABEL: faddp_2xhalf_commute:
 ; CHECKNOFP16:       // %bb.0: // %entry
 ; CHECKNOFP16-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECKNOFP16-NEXT:    dup v1.4h, v0.h[1]
 ; CHECKNOFP16-NEXT:    fcvtl v1.4s, v1.4h
 ; CHECKNOFP16-NEXT:    fcvtl v0.4s, v0.4h
 ; CHECKNOFP16-NEXT:    fadd v0.4s, v1.4s, v0.4s
 ; CHECKNOFP16-NEXT:    fcvtn v0.4h, v0.4s
 ; CHECKNOFP16-NEXT:    // kill: def $h0 killed $h0 killed $q0
 ; CHECKNOFP16-NEXT:    ret
 entry:
   %shift = shufflevector <2 x half> %a, <2 x half> undef, <2 x i32> <i32 1, i32 undef>
   %0 = fadd <2 x half> %shift, %a
   %1 = extractelement <2 x half> %0, i32 0
   ret half %1
 }
 
 define half @faddp_4xhalf(<4 x half> %a) {
 ; CHECK-LABEL: faddp_4xhalf:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    dup v1.4h, v0.h[1]
-; CHECK-NEXT:    fadd v0.4h, v0.4h, v1.4h
-; CHECK-NEXT:    // kill: def $h0 killed $h0 killed $q0
+; CHECK-NEXT:    faddp h0, v0.2h
 ; CHECK-NEXT:    ret
 ;
 ; CHECKNOFP16-LABEL: faddp_4xhalf:
 ; CHECKNOFP16:       // %bb.0: // %entry
 ; CHECKNOFP16-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECKNOFP16-NEXT:    dup v1.4h, v0.h[1]
 ; CHECKNOFP16-NEXT:    fcvtl v1.4s, v1.4h
 ; CHECKNOFP16-NEXT:    fcvtl v0.4s, v0.4h
 ; CHECKNOFP16-NEXT:    fadd v0.4s, v0.4s, v1.4s
 ; CHECKNOFP16-NEXT:    fcvtn v0.4h, v0.4s
 ; CHECKNOFP16-NEXT:    // kill: def $h0 killed $h0 killed $q0
 ; CHECKNOFP16-NEXT:    ret
 entry:
   %shift = shufflevector <4 x half> %a, <4 x half> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
   %0 = fadd <4 x half> %a, %shift
   %1 = extractelement <4 x half> %0, i32 0
   ret half %1
 }
 
 define half @faddp_4xhalf_commute(<4 x half> %a) {
 ; CHECK-LABEL: faddp_4xhalf_commute:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    dup v1.4h, v0.h[1]
-; CHECK-NEXT:    fadd v0.4h, v1.4h, v0.4h
-; CHECK-NEXT:    // kill: def $h0 killed $h0 killed $q0
+; CHECK-NEXT:    faddp h0, v0.2h
 ; CHECK-NEXT:    ret
 ;
 ; CHECKNOFP16-LABEL: faddp_4xhalf_commute:
 ; CHECKNOFP16:       // %bb.0: // %entry
 ; CHECKNOFP16-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECKNOFP16-NEXT:    dup v1.4h, v0.h[1]
 ; CHECKNOFP16-NEXT:    fcvtl v1.4s, v1.4h
 ; CHECKNOFP16-NEXT:    fcvtl v0.4s, v0.4h
 ; CHECKNOFP16-NEXT:    fadd v0.4s, v1.4s, v0.4s
 ; CHECKNOFP16-NEXT:    fcvtn v0.4h, v0.4s
 ; CHECKNOFP16-NEXT:    // kill: def $h0 killed $h0 killed $q0
 ; CHECKNOFP16-NEXT:    ret
 entry:
   %shift = shufflevector <4 x half> %a, <4 x half> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
   %0 = fadd <4 x half> %shift, %a
   %1 = extractelement <4 x half> %0, i32 0
   ret half %1
 }
 
 define half @faddp_8xhalf(<8 x half> %a) {
 ; CHECK-LABEL: faddp_8xhalf:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    dup v1.8h, v0.h[1]
-; CHECK-NEXT:    fadd v0.8h, v0.8h, v1.8h
-; CHECK-NEXT:    // kill: def $h0 killed $h0 killed $q0
+; CHECK-NEXT:    faddp h0, v0.2h
 ; CHECK-NEXT:    ret
 ;
 ; CHECKNOFP16-LABEL: faddp_8xhalf:
 ; CHECKNOFP16:       // %bb.0: // %entry
 ; CHECKNOFP16-NEXT:    dup v1.8h, v0.h[1]
 ; CHECKNOFP16-NEXT:    fcvt s0, h0
 ; CHECKNOFP16-NEXT:    fcvt s1, h1
 ; CHECKNOFP16-NEXT:    fadd s0, s0, s1
 ; CHECKNOFP16-NEXT:    fcvt h0, s0
 ; CHECKNOFP16-NEXT:    ret
 entry:
   %shift = shufflevector <8 x half> %a, <8 x half> undef, <8 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   %0 = fadd <8 x half> %a, %shift
   %1 = extractelement <8 x half> %0, i32 0
   ret half %1
 }
 
 define half @faddp_8xhalf_commute(<8 x half> %a) {
 ; CHECK-LABEL: faddp_8xhalf_commute:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    dup v1.8h, v0.h[1]
-; CHECK-NEXT:    fadd v0.8h, v1.8h, v0.8h
-; CHECK-NEXT:    // kill: def $h0 killed $h0 killed $q0
+; CHECK-NEXT:    faddp h0, v0.2h
 ; CHECK-NEXT:    ret
 ;
 ; CHECKNOFP16-LABEL: faddp_8xhalf_commute:
 ; CHECKNOFP16:       // %bb.0: // %entry
 ; CHECKNOFP16-NEXT:    dup v1.8h, v0.h[1]
 ; CHECKNOFP16-NEXT:    fcvt s0, h0
 ; CHECKNOFP16-NEXT:    fcvt s1, h1
 ; CHECKNOFP16-NEXT:    fadd s0, s1, s0
 ; CHECKNOFP16-NEXT:    fcvt h0, s0
 ; CHECKNOFP16-NEXT:    ret
 entry:
   %shift = shufflevector <8 x half> %a, <8 x half> undef, <8 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   %0 = fadd <8 x half> %shift, %a
   %1 = extractelement <8 x half> %0, i32 0
   ret half %1
 }

llvm/test/CodeGen/AArch64/faddp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc --mtriple aarch64 < %s | FileCheck %s
 
 define float @faddp_2xfloat(<2 x float> %a) {
 ; CHECK-LABEL: faddp_2xfloat:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    dup v1.2s, v0.s[1]
-; CHECK-NEXT:    fadd v0.2s, v0.2s, v1.2s
-; CHECK-NEXT:    // kill: def $s0 killed $s0 killed $q0
+; CHECK-NEXT:    faddp s0, v0.2s
 ; CHECK-NEXT:    ret
 entry:
   %shift = shufflevector <2 x float> %a, <2 x float> undef, <2 x i32> <i32 1, i32 undef>
   %0 = fadd <2 x float> %a, %shift
   %1 = extractelement <2 x float> %0, i32 0
   ret float %1
 }
 
 define float @faddp_4xfloat(<4 x float> %a) {
 ; CHECK-LABEL: faddp_4xfloat:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    dup v1.4s, v0.s[1]
-; CHECK-NEXT:    fadd v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    // kill: def $s0 killed $s0 killed $q0
+; CHECK-NEXT:    faddp s0, v0.2s
 ; CHECK-NEXT:    ret
 entry:
   %shift = shufflevector <4 x float> %a, <4 x float> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
   %0 = fadd <4 x float> %a, %shift
   %1 = extractelement <4 x float> %0, i32 0
   ret float %1
 }
 
 define float @faddp_4xfloat_commute(<4 x float> %a) {
 ; CHECK-LABEL: faddp_4xfloat_commute:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    dup v1.4s, v0.s[1]
-; CHECK-NEXT:    fadd v0.4s, v1.4s, v0.4s
-; CHECK-NEXT:    // kill: def $s0 killed $s0 killed $q0
+; CHECK-NEXT:    faddp s0, v0.2s
 ; CHECK-NEXT:    ret
 entry:
   %shift = shufflevector <4 x float> %a, <4 x float> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
   %0 = fadd <4 x float> %shift, %a
   %1 = extractelement <4 x float> %0, i32 0
   ret float %1
 }
 
 define float @faddp_2xfloat_commute(<2 x float> %a) {
 ; CHECK-LABEL: faddp_2xfloat_commute:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    dup v1.2s, v0.s[1]
-; CHECK-NEXT:    fadd v0.2s, v1.2s, v0.2s
-; CHECK-NEXT:    // kill: def $s0 killed $s0 killed $q0
+; CHECK-NEXT:    faddp s0, v0.2s
 ; CHECK-NEXT:    ret
 entry:
   %shift = shufflevector <2 x float> %a, <2 x float> undef, <2 x i32> <i32 1, i32 undef>
   %0 = fadd <2 x float> %shift, %a
   %1 = extractelement <2 x float> %0, i32 0
   ret float %1
 }
 
 define double @faddp_2xdouble(<2 x double> %a) {
 ; CHECK-LABEL: faddp_2xdouble:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    dup v1.2d, v0.d[1]
-; CHECK-NEXT:    fadd v0.2d, v0.2d, v1.2d
-; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; CHECK-NEXT:    faddp d0, v0.2d
 ; CHECK-NEXT:    ret
 entry:
   %shift = shufflevector <2 x double> %a, <2 x double> undef, <2 x i32> <i32 1, i32 undef>
   %0 = fadd <2 x double> %a, %shift
   %1 = extractelement <2 x double> %0, i32 0
   ret double %1
 }
 
 define double @faddp_2xdouble_commute(<2 x double> %a) {
 ; CHECK-LABEL: faddp_2xdouble_commute:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    dup v1.2d, v0.d[1]
-; CHECK-NEXT:    fadd v0.2d, v1.2d, v0.2d
-; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; CHECK-NEXT:    faddp d0, v0.2d
 ; CHECK-NEXT:    ret
 entry:
   %shift = shufflevector <2 x double> %a, <2 x double> undef, <2 x i32> <i32 1, i32 undef>
   %0 = fadd <2 x double> %shift, %a
   %1 = extractelement <2 x double> %0, i32 0
   ret double %1
 }
 
 define i64 @addp_2xi64(<2 x i64> %a) {
 ; CHECK-LABEL: addp_2xi64:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    dup v1.2d, v0.d[1]
-; CHECK-NEXT:    add v0.2d, v0.2d, v1.2d
+; CHECK-NEXT:    addp d0, v0.2d
 ; CHECK-NEXT:    fmov x0, d0
 ; CHECK-NEXT:    ret
 entry:
   %shift = shufflevector <2 x i64> %a, <2 x i64> undef, <2 x i32> <i32 1, i32 undef>
   %0 = add <2 x i64> %a, %shift
   %1 = extractelement <2 x i64> %0, i32 0
   ret i64 %1
 }
 
 define i64 @addp_2xi64_commute(<2 x i64> %a) {
 ; CHECK-LABEL: addp_2xi64_commute:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    dup v1.2d, v0.d[1]
-; CHECK-NEXT:    add v0.2d, v1.2d, v0.2d
+; CHECK-NEXT:    addp d0, v0.2d
 ; CHECK-NEXT:    fmov x0, d0
 ; CHECK-NEXT:    ret
 entry:
   %shift = shufflevector <2 x i64> %a, <2 x i64> undef, <2 x i32> <i32 1, i32 undef>
   %0 = add <2 x i64> %shift, %a
   %1 = extractelement <2 x i64> %0, i32 0
   ret i64 %1
 }

llvm/test/CodeGen/AArch64/vecreduce-fadd.ll

; CHECKNOFP16-NEXT:    ret
   %r = call fast float @llvm.experimental.vector.reduce.v2.fadd.f32.v2f32(float 0.0, <2 x float> %bin.rdx)
   ret float %r
 }
 
 define half @add_HalfH(<4 x half> %bin.rdx)  {
 ; CHECK-LABEL: add_HalfH:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
-; CHECK-NEXT:    mov h3, v0.h[1]
 ; CHECK-NEXT:    mov h1, v0.h[3]
 ; CHECK-NEXT:    mov h2, v0.h[2]
-; CHECK-NEXT:    fadd h0, h0, h3
+; CHECK-NEXT:    faddp h0, v0.2h
 ; CHECK-NEXT:    fadd h0, h0, h2
 ; CHECK-NEXT:    fadd h0, h0, h1
 ; CHECK-NEXT:    ret
 ;
 ; CHECKNOFP16-LABEL: add_HalfH:
 ; CHECKNOFP16:       // %bb.0:
 ; CHECKNOFP16-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECKNOFP16-NEXT:    mov h3, v0.h[1]

 }
 
 
 define half @add_H(<8 x half> %bin.rdx)  {
 ; CHECK-LABEL: add_H:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
 ; CHECK-NEXT:    fadd v0.4h, v0.4h, v1.4h
-; CHECK-NEXT:    mov h1, v0.h[1]
-; CHECK-NEXT:    mov h2, v0.h[2]
-; CHECK-NEXT:    fadd h1, h0, h1
-; CHECK-NEXT:    fadd h1, h1, h2
+; CHECK-NEXT:    mov h1, v0.h[2]
+; CHECK-NEXT:    faddp h2, v0.2h
+; CHECK-NEXT:    fadd h1, h2, h1
 ; CHECK-NEXT:    mov h0, v0.h[3]
 ; CHECK-NEXT:    fadd h0, h1, h0
 ; CHECK-NEXT:    ret
 ;
 ; CHECKNOFP16-LABEL: add_H:
 ; CHECKNOFP16:       // %bb.0:
 ; CHECKNOFP16-NEXT:    mov h7, v0.h[1]
 ; CHECKNOFP16-NEXT:    mov h1, v0.h[7]

 ; CHECKNOFP16-NEXT:    fcvt s0, h0
 ; CHECKNOFP16-NEXT:    fadd s0, s0, s2
 ; CHECKNOFP16-NEXT:    fcvt h0, s0
 ; CHECKNOFP16-NEXT:    fcvt s0, h0
 ; CHECKNOFP16-NEXT:    fcvt s1, h1
 ; CHECKNOFP16-NEXT:    fadd s0, s0, s1
 ; CHECKNOFP16-NEXT:    fcvt h0, s0
 ; CHECKNOFP16-NEXT:    ret
-
   %r = call fast half @llvm.experimental.vector.reduce.v2.fadd.f16.v8f16(half 0.0, <8 x half> %bin.rdx)
   ret half %r
 }
 
 define float @add_S(<4 x float> %bin.rdx)  {
 ; CHECK-LABEL: add_S:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8

 }
 
 define half @add_2H(<16 x half> %bin.rdx)  {
 ; CHECK-LABEL: add_2H:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fadd v0.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
 ; CHECK-NEXT:    fadd v0.4h, v0.4h, v1.4h
-; CHECK-NEXT:    mov h1, v0.h[1]
-; CHECK-NEXT:    mov h2, v0.h[2]
-; CHECK-NEXT:    fadd h1, h0, h1
-; CHECK-NEXT:    fadd h1, h1, h2
+; CHECK-NEXT:    mov h1, v0.h[2]
+; CHECK-NEXT:    faddp h2, v0.2h
+; CHECK-NEXT:    fadd h1, h2, h1
 ; CHECK-NEXT:    mov h0, v0.h[3]
 ; CHECK-NEXT:    fadd h0, h1, h0
 ; CHECK-NEXT:    ret
 ;
 ; CHECKNOFP16-LABEL: add_2H:
 ; CHECKNOFP16:       // %bb.0:
 ; CHECKNOFP16-NEXT:    mov h2, v1.h[1]
 ; CHECKNOFP16-NEXT:    mov h3, v0.h[1]