[CostModel][AArch64] Improve cost model for vector reduction intrinsics

[RosieSumpter]

Summary

OR, XOR and AND entries are added to the cost table. An extra cost
is added when vector splitting occurs.

This is done to address the issue of a missed SLP vectorization
opportunity due to unreasonably high costs being attributed to the vector
Or reduction (see: https://bugs.llvm.org/show_bug.cgi?id=44593).

[dmgreen]

Do And and Xor reductions work in the same way, with the same costs? Can we do those at the same time too?

[RosieSumpter]

• Added XOR and AND entries to the table and their cases (which fall through to the code for OR)

• Modified the reduce-and.ll test, and created a reduce-xor.ll test

[SjoerdMeijer]

Nice one. This work was motivated by a missed opportunity in the SLP vectoriser.
Can you separately create a reduced test case for that, so that we can first precommit that?
Then, that tests should be modified here.

[SjoerdMeijer]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1793

You can say something here how this looks like for these new opcodes that we are adding.

I would also appreciate a pointer to the codegen test so that's easy to verify this. I was going to say that a cost of 15 looks high, but I was too lazy to find the codegen test. :)

1829

Nit: to reduce indentation and increase readability, can you return early here:

const auto *Entry = CostTableLookup(CostTblNoPairwise, ISD, MTy);
if (!Entry)
  break;
...

1830

If this cast can fail, we are accessing a null pointer below?

[RosieSumpter]

In D104538#2830246, @SjoerdMeijer wrote:

Can you separately create a reduced test case for that, so that we can first precommit that?

Sure, will do that

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1793

There is a codegen test for i1 types (test/CodeGen/AArch64/vecreduce-bool.ll) but I couldn't find a test for the other vector types. These values are just from a manual check I did of the codegen (e.g. llc reduce-or.ll -mtriple=aarch64)

1830

I don't think a failed cast would make it this far since the cast calls assert() doesn't it? If it still needs checking, could do an if(ValVTy) before the following code?

[RosieSumpter]

The slp-or-reduction.ll test (https://reviews.llvm.org/rG2251f33bef38) is updated here and shows that the SLP vectorizer is now invoked.

[RosieSumpter]

Updates have been made to the XOR and AND SLP vectorization regression tests. They now show that vectorization occurs with this cost model change.

[SjoerdMeijer]

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

1793

In that case, would you mind adding those codegen tests (separately)?
You can then refer to that here in a comment.

1830

Ah yes, I read this as a dyn_cast, but it's a cast, so is correct.

[RosieSumpter]

• Added v2i32 costs to table

• Added v2i32 to cost model tests

• Added comment referring to codegen tests

• Break early if there's no table entry

[SjoerdMeijer]

Thanks, this looks like a good improvement to me.

llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp

AArch64TTIImpl::getArithmeticReductionCost(unsigned Opcode, VectorType *ValTy,
 
   std::pair<InstructionCost, MVT> LT = TLI->getTypeLegalizationCost(DL, ValTy);
   MVT MTy = LT.second;
   int ISD = TLI->InstructionOpcodeToISD(Opcode);
   assert(ISD && "Invalid opcode");
 
   // Horizontal adds can use the 'addv' instruction. We model the cost of these
   // instructions as normal vector adds. This is the only arithmetic vector
   // reduction operation for which we have an instruction.
   static const CostTblEntry CostTblNoPairwise[]{
-      {ISD::ADD, MVT::v8i8,  1},
-      {ISD::ADD, MVT::v16i8, 1},
-      {ISD::ADD, MVT::v4i16, 1},
-      {ISD::ADD, MVT::v8i16, 1},
-      {ISD::ADD, MVT::v4i32, 1},
+      {ISD::ADD, MVT::v8i8,   1},
+      {ISD::ADD, MVT::v16i8,  1},
+      {ISD::ADD, MVT::v4i16,  1},
+      {ISD::ADD, MVT::v8i16,  1},
+      {ISD::ADD, MVT::v4i32,  1},
+      {ISD::OR,  MVT::v8i8,  15},
+      {ISD::OR,  MVT::v16i8, 17},
+      {ISD::OR,  MVT::v4i16,  7},
+      {ISD::OR,  MVT::v8i16,  9},
+      {ISD::OR,  MVT::v4i32,  5},
+      {ISD::OR,  MVT::v2i64,  3},
+      {ISD::XOR, MVT::v8i8,  15},
+      {ISD::XOR, MVT::v16i8, 17},
+      {ISD::XOR, MVT::v4i16,  7},
+      {ISD::XOR, MVT::v8i16,  9},
+      {ISD::XOR, MVT::v4i32,  5},
+      {ISD::XOR, MVT::v2i64,  3},
+      {ISD::AND, MVT::v8i8,  15},
+      {ISD::AND, MVT::v16i8, 17},
+      {ISD::AND, MVT::v4i16,  7},
+      {ISD::AND, MVT::v8i16,  9},
+      {ISD::AND, MVT::v4i32,  5},
+      {ISD::AND, MVT::v2i64,  3},
   };
-
+  switch (ISD) {
+  default:
+    break;
+  case ISD::ADD:
     if (const auto *Entry = CostTableLookup(CostTblNoPairwise, ISD, MTy))
       return LT.first * Entry->Cost;
-
+    break;
+  case ISD::XOR:
+  case ISD::AND:
+  case ISD::OR:
+    if (const auto *Entry = CostTableLookup(CostTblNoPairwise, ISD, MTy)) {
+      auto *ValVTy = cast<FixedVectorType>(ValTy);
+      if (!ValVTy->getElementType()->isIntegerTy(1) &&
+          MTy.getVectorNumElements() <= ValVTy->getNumElements() &&
+          isPowerOf2_32(ValVTy->getNumElements())) {
+        InstructionCost ExtraCost = 0;
+        if (LT.first != 1) {
+          // Type needs to be split, so there is an extra cost of LT.first - 1
+          // arithmetic ops.
+          auto *Ty = FixedVectorType::get(ValTy->getElementType(),
+                                          MTy.getVectorNumElements());
+          ExtraCost = getArithmeticInstrCost(Opcode, Ty, CostKind);
+          ExtraCost *= LT.first - 1;
+        }
+        return Entry->Cost + ExtraCost;
+      }
+    }
+    break;
+  }
   return BaseT::getArithmeticReductionCost(Opcode, ValTy, IsPairwiseForm,
                                            CostKind);
 }
 
 InstructionCost AArch64TTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
                                                VectorType *Tp,
                                                ArrayRef<int> Mask, int Index,
                                                VectorType *SubTp) {

llvm/test/Analysis/CostModel/AArch64/reduce-and.ll

 ; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
 ; RUN: opt < %s -mtriple=aarch64-unknown-linux-gnu -cost-model -cost-kind=throughput -analyze | FileCheck %s
 
 define i32 @reduce_i1(i32 %arg) {
 ; CHECK-LABEL: 'reduce_i1'
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %V1 = call i1 @llvm.vector.reduce.and.v1i1(<1 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V2 = call i1 @llvm.vector.reduce.and.v2i1(<2 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V4 = call i1 @llvm.vector.reduce.and.v4i1(<4 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V8 = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V16 = call i1 @llvm.vector.reduce.and.v16i1(<16 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 91 for instruction: %V32 = call i1 @llvm.vector.reduce.and.v32i1(<32 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 181 for instruction: %V64 = call i1 @llvm.vector.reduce.and.v64i1(<64 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 362 for instruction: %V128 = call i1 @llvm.vector.reduce.and.v128i1(<128 x i1> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %V1i8 = call i8 @llvm.vector.reduce.and.v1i8(<1 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 13 for instruction: %V3i8 = call i8 @llvm.vector.reduce.and.v3i8(<3 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 7 for instruction: %V4i8 = call i8 @llvm.vector.reduce.and.v4i8(<4 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 15 for instruction: %V8i8 = call i8 @llvm.vector.reduce.and.v8i8(<8 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 17 for instruction: %V16i8 = call i8 @llvm.vector.reduce.and.v16i8(<16 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 18 for instruction: %V32i8 = call i8 @llvm.vector.reduce.and.v32i8(<32 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 20 for instruction: %V64i8 = call i8 @llvm.vector.reduce.and.v64i8(<64 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 7 for instruction: %V4i16 = call i16 @llvm.vector.reduce.and.v4i16(<4 x i16> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 9 for instruction: %V8i16 = call i16 @llvm.vector.reduce.and.v8i16(<8 x i16> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 10 for instruction: %V16i16 = call i16 @llvm.vector.reduce.and.v16i16(<16 x i16> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 5 for instruction: %V4i32 = call i32 @llvm.vector.reduce.and.v4i32(<4 x i32> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %V8i32 = call i32 @llvm.vector.reduce.and.v8i32(<8 x i32> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %V2i64 = call i64 @llvm.vector.reduce.and.v2i64(<2 x i64> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %V4i64 = call i64 @llvm.vector.reduce.and.v4i64(<4 x i64> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret i32 undef
-;
+
   %V1   = call i1 @llvm.vector.reduce.and.v1i1(<1 x i1> undef)
   %V2   = call i1 @llvm.vector.reduce.and.v2i1(<2 x i1> undef)
   %V4   = call i1 @llvm.vector.reduce.and.v4i1(<4 x i1> undef)
   %V8   = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> undef)
   %V16  = call i1 @llvm.vector.reduce.and.v16i1(<16 x i1> undef)
   %V32  = call i1 @llvm.vector.reduce.and.v32i1(<32 x i1> undef)
   %V64  = call i1 @llvm.vector.reduce.and.v64i1(<64 x i1> undef)
   %V128 = call i1 @llvm.vector.reduce.and.v128i1(<128 x i1> undef)
+
+  %V1i8 = call i8 @llvm.vector.reduce.and.v1i8(<1 x i8> undef)
+  %V3i8 = call i8 @llvm.vector.reduce.and.v3i8(<3 x i8> undef)
+  %V4i8 = call i8 @llvm.vector.reduce.and.v4i8(<4 x i8> undef)
+  %V8i8 = call i8 @llvm.vector.reduce.and.v8i8(<8 x i8> undef)
+  %V16i8 = call i8 @llvm.vector.reduce.and.v16i8(<16 x i8> undef)
+  %V32i8 = call i8 @llvm.vector.reduce.and.v32i8(<32 x i8> undef)
+  %V64i8 = call i8 @llvm.vector.reduce.and.v64i8(<64 x i8> undef)
+  %V4i16 = call i16 @llvm.vector.reduce.and.v4i16(<4 x i16> undef)
+  %V8i16 = call i16 @llvm.vector.reduce.and.v8i16(<8 x i16> undef)
+  %V16i16 = call i16 @llvm.vector.reduce.and.v16i16(<16 x i16> undef)
+  %V4i32 = call i32 @llvm.vector.reduce.and.v4i32(<4 x i32> undef)
+  %V8i32 = call i32 @llvm.vector.reduce.and.v8i32(<8 x i32> undef)
+  %V2i64 = call i64 @llvm.vector.reduce.and.v2i64(<2 x i64> undef)
+  %V4i64 = call i64 @llvm.vector.reduce.and.v4i64(<4 x i64> undef)
   ret i32 undef
 }
 
 declare i1 @llvm.vector.reduce.and.v1i1(<1 x i1>)
 declare i1 @llvm.vector.reduce.and.v2i1(<2 x i1>)
 declare i1 @llvm.vector.reduce.and.v4i1(<4 x i1>)
 declare i1 @llvm.vector.reduce.and.v8i1(<8 x i1>)
 declare i1 @llvm.vector.reduce.and.v16i1(<16 x i1>)
 declare i1 @llvm.vector.reduce.and.v32i1(<32 x i1>)
 declare i1 @llvm.vector.reduce.and.v64i1(<64 x i1>)
 declare i1 @llvm.vector.reduce.and.v128i1(<128 x i1>)
+declare i8 @llvm.vector.reduce.and.v1i8(<1 x i8>)
+declare i8 @llvm.vector.reduce.and.v3i8(<3 x i8>)
+declare i8 @llvm.vector.reduce.and.v4i8(<4 x i8>)
+declare i8 @llvm.vector.reduce.and.v8i8(<8 x i8>)
+declare i8 @llvm.vector.reduce.and.v16i8(<16 x i8>)
+declare i8 @llvm.vector.reduce.and.v32i8(<32 x i8>)
+declare i8 @llvm.vector.reduce.and.v64i8(<64 x i8>)
+declare i16 @llvm.vector.reduce.and.v4i16(<4 x i16>)
+declare i16 @llvm.vector.reduce.and.v8i16(<8 x i16>)
+declare i16 @llvm.vector.reduce.and.v16i16(<16 x i16>)
+declare i32 @llvm.vector.reduce.and.v4i32(<4 x i32>)
+declare i32 @llvm.vector.reduce.and.v8i32(<8 x i32>)
+declare i64 @llvm.vector.reduce.and.v2i64(<2 x i64>)
+declare i64 @llvm.vector.reduce.and.v4i64(<4 x i64>)

llvm/test/Analysis/CostModel/AArch64/reduce-or.ll

 ; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
 ; RUN: opt < %s -mtriple=aarch64-unknown-linux-gnu -cost-model -cost-kind=throughput -analyze | FileCheck %s
 
 define i32 @reduce_i1(i32 %arg) {
 ; CHECK-LABEL: 'reduce_i1'
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %V1 = call i1 @llvm.vector.reduce.or.v1i1(<1 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V2 = call i1 @llvm.vector.reduce.or.v2i1(<2 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V4 = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V8 = call i1 @llvm.vector.reduce.or.v8i1(<8 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V16 = call i1 @llvm.vector.reduce.or.v16i1(<16 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 91 for instruction: %V32 = call i1 @llvm.vector.reduce.or.v32i1(<32 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 181 for instruction: %V64 = call i1 @llvm.vector.reduce.or.v64i1(<64 x i1> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 362 for instruction: %V128 = call i1 @llvm.vector.reduce.or.v128i1(<128 x i1> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %V1i8 = call i8 @llvm.vector.reduce.or.v1i8(<1 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 13 for instruction: %V3i8 = call i8 @llvm.vector.reduce.or.v3i8(<3 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 7 for instruction: %V4i8 = call i8 @llvm.vector.reduce.or.v4i8(<4 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 15 for instruction: %V8i8 = call i8 @llvm.vector.reduce.or.v8i8(<8 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 17 for instruction: %V16i8 = call i8 @llvm.vector.reduce.or.v16i8(<16 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 18 for instruction: %V32i8 = call i8 @llvm.vector.reduce.or.v32i8(<32 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 20 for instruction: %V64i8 = call i8 @llvm.vector.reduce.or.v64i8(<64 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 7 for instruction: %V4i16 = call i16 @llvm.vector.reduce.or.v4i16(<4 x i16> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 9 for instruction: %V8i16 = call i16 @llvm.vector.reduce.or.v8i16(<8 x i16> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 10 for instruction: %V16i16 = call i16 @llvm.vector.reduce.or.v16i16(<16 x i16> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 5 for instruction: %V4i32 = call i32 @llvm.vector.reduce.or.v4i32(<4 x i32> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %V8i32 = call i32 @llvm.vector.reduce.or.v8i32(<8 x i32> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %V2i64 = call i64 @llvm.vector.reduce.or.v2i64(<2 x i64> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %V4i64 = call i64 @llvm.vector.reduce.or.v4i64(<4 x i64> undef)
 ; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret i32 undef
-;
+
   %V1   = call i1 @llvm.vector.reduce.or.v1i1(<1 x i1> undef)
   %V2   = call i1 @llvm.vector.reduce.or.v2i1(<2 x i1> undef)
   %V4   = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> undef)
   %V8   = call i1 @llvm.vector.reduce.or.v8i1(<8 x i1> undef)
   %V16  = call i1 @llvm.vector.reduce.or.v16i1(<16 x i1> undef)
   %V32  = call i1 @llvm.vector.reduce.or.v32i1(<32 x i1> undef)
   %V64  = call i1 @llvm.vector.reduce.or.v64i1(<64 x i1> undef)
   %V128 = call i1 @llvm.vector.reduce.or.v128i1(<128 x i1> undef)
+
+  %V1i8 = call i8 @llvm.vector.reduce.or.v1i8(<1 x i8> undef)
+  %V3i8 = call i8 @llvm.vector.reduce.or.v3i8(<3 x i8> undef)
+  %V4i8 = call i8 @llvm.vector.reduce.or.v4i8(<4 x i8> undef)
+  %V8i8 = call i8 @llvm.vector.reduce.or.v8i8(<8 x i8> undef)
+  %V16i8 = call i8 @llvm.vector.reduce.or.v16i8(<16 x i8> undef)
+  %V32i8 = call i8 @llvm.vector.reduce.or.v32i8(<32 x i8> undef)
+  %V64i8 = call i8 @llvm.vector.reduce.or.v64i8(<64 x i8> undef)
+  %V4i16 = call i16 @llvm.vector.reduce.or.v4i16(<4 x i16> undef)
+  %V8i16 = call i16 @llvm.vector.reduce.or.v8i16(<8 x i16> undef)
+  %V16i16 = call i16 @llvm.vector.reduce.or.v16i16(<16 x i16> undef)
+  %V4i32 = call i32 @llvm.vector.reduce.or.v4i32(<4 x i32> undef)
+  %V8i32 = call i32 @llvm.vector.reduce.or.v8i32(<8 x i32> undef)
+  %V2i64 = call i64 @llvm.vector.reduce.or.v2i64(<2 x i64> undef)
+  %V4i64 = call i64 @llvm.vector.reduce.or.v4i64(<4 x i64> undef)
   ret i32 undef
 }
 
 declare i1 @llvm.vector.reduce.or.v1i1(<1 x i1>)
 declare i1 @llvm.vector.reduce.or.v2i1(<2 x i1>)
 declare i1 @llvm.vector.reduce.or.v4i1(<4 x i1>)
 declare i1 @llvm.vector.reduce.or.v8i1(<8 x i1>)
 declare i1 @llvm.vector.reduce.or.v16i1(<16 x i1>)
 declare i1 @llvm.vector.reduce.or.v32i1(<32 x i1>)
 declare i1 @llvm.vector.reduce.or.v64i1(<64 x i1>)
 declare i1 @llvm.vector.reduce.or.v128i1(<128 x i1>)
+declare i8 @llvm.vector.reduce.or.v1i8(<1 x i8>)
+declare i8 @llvm.vector.reduce.or.v3i8(<3 x i8>)
+declare i8 @llvm.vector.reduce.or.v4i8(<4 x i8>)
+declare i8 @llvm.vector.reduce.or.v8i8(<8 x i8>)
+declare i8 @llvm.vector.reduce.or.v16i8(<16 x i8>)
+declare i8 @llvm.vector.reduce.or.v32i8(<32 x i8>)
+declare i8 @llvm.vector.reduce.or.v64i8(<64 x i8>)
+declare i16 @llvm.vector.reduce.or.v4i16(<4 x i16>)
+declare i16 @llvm.vector.reduce.or.v8i16(<8 x i16>)
+declare i16 @llvm.vector.reduce.or.v16i16(<16 x i16>)
+declare i32 @llvm.vector.reduce.or.v4i32(<4 x i32>)
+declare i32 @llvm.vector.reduce.or.v8i32(<8 x i32>)
+declare i64 @llvm.vector.reduce.or.v2i64(<2 x i64>)
+declare i64 @llvm.vector.reduce.or.v4i64(<4 x i64>)

llvm/test/Analysis/CostModel/AArch64/reduce-xor.ll

+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
+; RUN: opt < %s -mtriple=aarch64-unknown-linux-gnu -cost-model -cost-kind=throughput -analyze | FileCheck %s
+
+define i32 @reduce_i1(i32 %arg) {
+; CHECK-LABEL: 'reduce_i1'
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %V1 = call i1 @llvm.vector.reduce.xor.v1i1(<1 x i1> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V2 = call i1 @llvm.vector.reduce.xor.v2i1(<2 x i1> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 38 for instruction: %V4 = call i1 @llvm.vector.reduce.xor.v4i1(<4 x i1> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 129 for instruction: %V8 = call i1 @llvm.vector.reduce.xor.v8i1(<8 x i1> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 364 for instruction: %V16 = call i1 @llvm.vector.reduce.xor.v16i1(<16 x i1> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 455 for instruction: %V32 = call i1 @llvm.vector.reduce.xor.v32i1(<32 x i1> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 637 for instruction: %V64 = call i1 @llvm.vector.reduce.xor.v64i1(<64 x i1> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1001 for instruction: %V128 = call i1 @llvm.vector.reduce.xor.v128i1(<128 x i1> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: %V1i8 = call i8 @llvm.vector.reduce.xor.v1i8(<1 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 13 for instruction: %V3i8 = call i8 @llvm.vector.reduce.xor.v3i8(<3 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 7 for instruction: %V4i8 = call i8 @llvm.vector.reduce.xor.v4i8(<4 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 15 for instruction: %V8i8 = call i8 @llvm.vector.reduce.xor.v8i8(<8 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 17 for instruction: %V16i8 = call i8 @llvm.vector.reduce.xor.v16i8(<16 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 18 for instruction: %V32i8 = call i8 @llvm.vector.reduce.xor.v32i8(<32 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 20 for instruction: %V64i8 = call i8 @llvm.vector.reduce.xor.v64i8(<64 x i8> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 7 for instruction: %V4i16 = call i16 @llvm.vector.reduce.xor.v4i16(<4 x i16> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 9 for instruction: %V8i16 = call i16 @llvm.vector.reduce.xor.v8i16(<8 x i16> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 10 for instruction: %V16i16 = call i16 @llvm.vector.reduce.xor.v16i16(<16 x i16> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 5 for instruction: %V4i32 = call i32 @llvm.vector.reduce.xor.v4i32(<4 x i32> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 6 for instruction: %V8i32 = call i32 @llvm.vector.reduce.xor.v8i32(<8 x i32> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 3 for instruction: %V2i64 = call i64 @llvm.vector.reduce.xor.v2i64(<2 x i64> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %V4i64 = call i64 @llvm.vector.reduce.xor.v4i64(<4 x i64> undef)
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret i32 undef
+
+  %V1   = call i1 @llvm.vector.reduce.xor.v1i1(<1 x i1> undef)
+  %V2   = call i1 @llvm.vector.reduce.xor.v2i1(<2 x i1> undef)
+  %V4   = call i1 @llvm.vector.reduce.xor.v4i1(<4 x i1> undef)
+  %V8   = call i1 @llvm.vector.reduce.xor.v8i1(<8 x i1> undef)
+  %V16  = call i1 @llvm.vector.reduce.xor.v16i1(<16 x i1> undef)
+  %V32  = call i1 @llvm.vector.reduce.xor.v32i1(<32 x i1> undef)
+  %V64  = call i1 @llvm.vector.reduce.xor.v64i1(<64 x i1> undef)
+  %V128 = call i1 @llvm.vector.reduce.xor.v128i1(<128 x i1> undef)
+
+  %V1i8 = call i8 @llvm.vector.reduce.xor.v1i8(<1 x i8> undef)
+  %V3i8 = call i8 @llvm.vector.reduce.xor.v3i8(<3 x i8> undef)
+  %V4i8 = call i8 @llvm.vector.reduce.xor.v4i8(<4 x i8> undef)
+  %V8i8 = call i8 @llvm.vector.reduce.xor.v8i8(<8 x i8> undef)
+  %V16i8 = call i8 @llvm.vector.reduce.xor.v16i8(<16 x i8> undef)
+  %V32i8 = call i8 @llvm.vector.reduce.xor.v32i8(<32 x i8> undef)
+  %V64i8 = call i8 @llvm.vector.reduce.xor.v64i8(<64 x i8> undef)
+  %V4i16 = call i16 @llvm.vector.reduce.xor.v4i16(<4 x i16> undef)
+  %V8i16 = call i16 @llvm.vector.reduce.xor.v8i16(<8 x i16> undef)
+  %V16i16 = call i16 @llvm.vector.reduce.xor.v16i16(<16 x i16> undef)
+  %V4i32 = call i32 @llvm.vector.reduce.xor.v4i32(<4 x i32> undef)
+  %V8i32 = call i32 @llvm.vector.reduce.xor.v8i32(<8 x i32> undef)
+  %V2i64 = call i64 @llvm.vector.reduce.xor.v2i64(<2 x i64> undef)
+  %V4i64 = call i64 @llvm.vector.reduce.xor.v4i64(<4 x i64> undef)
+  ret i32 undef
+}
+
+declare i1 @llvm.vector.reduce.xor.v1i1(<1 x i1>)
+declare i1 @llvm.vector.reduce.xor.v2i1(<2 x i1>)
+declare i1 @llvm.vector.reduce.xor.v4i1(<4 x i1>)
+declare i1 @llvm.vector.reduce.xor.v8i1(<8 x i1>)
+declare i1 @llvm.vector.reduce.xor.v16i1(<16 x i1>)
+declare i1 @llvm.vector.reduce.xor.v32i1(<32 x i1>)
+declare i1 @llvm.vector.reduce.xor.v64i1(<64 x i1>)
+declare i1 @llvm.vector.reduce.xor.v128i1(<128 x i1>)
+declare i8 @llvm.vector.reduce.xor.v1i8(<1 x i8>)
+declare i8 @llvm.vector.reduce.xor.v3i8(<3 x i8>)
+declare i8 @llvm.vector.reduce.xor.v4i8(<4 x i8>)
+declare i8 @llvm.vector.reduce.xor.v8i8(<8 x i8>)
+declare i8 @llvm.vector.reduce.xor.v16i8(<16 x i8>)
+declare i8 @llvm.vector.reduce.xor.v32i8(<32 x i8>)
+declare i8 @llvm.vector.reduce.xor.v64i8(<64 x i8>)
+declare i16 @llvm.vector.reduce.xor.v4i16(<4 x i16>)
+declare i16 @llvm.vector.reduce.xor.v8i16(<8 x i16>)
+declare i16 @llvm.vector.reduce.xor.v16i16(<16 x i16>)
+declare i32 @llvm.vector.reduce.xor.v4i32(<4 x i32>)
+declare i32 @llvm.vector.reduce.xor.v8i32(<8 x i32>)
+declare i64 @llvm.vector.reduce.xor.v2i64(<2 x i64>)
+declare i64 @llvm.vector.reduce.xor.v4i64(<4 x i64>)

llvm/test/Transforms/SLPVectorizer/AArch64/slp-and-reduction.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -S -mtriple=aarch64 -slp-vectorizer | FileCheck %s
 
 %struct.buf = type { [8 x i8] }
 
 define i8 @reduce_and(%struct.buf* %a, %struct.buf* %b) {
 ; CHECK-LABEL: @reduce_and(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_BUF:%.*]], %struct.buf* [[A:%.*]], i64 0, i32 0, i64 0
-; CHECK-NEXT:    [[TMP0:%.*]] = load i8, i8* [[ARRAYIDX]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B:%.*]], i64 0, i32 0, i64 0
-; CHECK-NEXT:    [[TMP1:%.*]] = load i8, i8* [[ARRAYIDX3]], align 1
-; CHECK-NEXT:    [[XOR12:%.*]] = xor i8 [[TMP1]], [[TMP0]]
-; CHECK-NEXT:    [[AND13:%.*]] = and i8 [[XOR12]], 1
 ; CHECK-NEXT:    [[ARRAYIDX_1:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 1
-; CHECK-NEXT:    [[TMP2:%.*]] = load i8, i8* [[ARRAYIDX_1]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_1:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 1
-; CHECK-NEXT:    [[TMP3:%.*]] = load i8, i8* [[ARRAYIDX3_1]], align 1
-; CHECK-NEXT:    [[XOR12_1:%.*]] = xor i8 [[TMP3]], [[TMP2]]
-; CHECK-NEXT:    [[AND13_1:%.*]] = and i8 [[XOR12_1]], [[AND13]]
 ; CHECK-NEXT:    [[ARRAYIDX_2:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 2
-; CHECK-NEXT:    [[TMP4:%.*]] = load i8, i8* [[ARRAYIDX_2]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_2:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 2
-; CHECK-NEXT:    [[TMP5:%.*]] = load i8, i8* [[ARRAYIDX3_2]], align 1
-; CHECK-NEXT:    [[XOR12_2:%.*]] = xor i8 [[TMP5]], [[TMP4]]
-; CHECK-NEXT:    [[AND13_2:%.*]] = and i8 [[XOR12_2]], [[AND13_1]]
 ; CHECK-NEXT:    [[ARRAYIDX_3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 3
-; CHECK-NEXT:    [[TMP6:%.*]] = load i8, i8* [[ARRAYIDX_3]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 3
-; CHECK-NEXT:    [[TMP7:%.*]] = load i8, i8* [[ARRAYIDX3_3]], align 1
-; CHECK-NEXT:    [[XOR12_3:%.*]] = xor i8 [[TMP7]], [[TMP6]]
-; CHECK-NEXT:    [[AND13_3:%.*]] = and i8 [[XOR12_3]], [[AND13_2]]
 ; CHECK-NEXT:    [[ARRAYIDX_4:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 4
-; CHECK-NEXT:    [[TMP8:%.*]] = load i8, i8* [[ARRAYIDX_4]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_4:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 4
-; CHECK-NEXT:    [[TMP9:%.*]] = load i8, i8* [[ARRAYIDX3_4]], align 1
-; CHECK-NEXT:    [[XOR12_4:%.*]] = xor i8 [[TMP9]], [[TMP8]]
-; CHECK-NEXT:    [[AND13_4:%.*]] = and i8 [[XOR12_4]], [[AND13_3]]
 ; CHECK-NEXT:    [[ARRAYIDX_5:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 5
-; CHECK-NEXT:    [[TMP10:%.*]] = load i8, i8* [[ARRAYIDX_5]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_5:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 5
-; CHECK-NEXT:    [[TMP11:%.*]] = load i8, i8* [[ARRAYIDX3_5]], align 1
-; CHECK-NEXT:    [[XOR12_5:%.*]] = xor i8 [[TMP11]], [[TMP10]]
-; CHECK-NEXT:    [[AND13_5:%.*]] = and i8 [[XOR12_5]], [[AND13_4]]
 ; CHECK-NEXT:    [[ARRAYIDX_6:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 6
-; CHECK-NEXT:    [[TMP12:%.*]] = load i8, i8* [[ARRAYIDX_6]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_6:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 6
-; CHECK-NEXT:    [[TMP13:%.*]] = load i8, i8* [[ARRAYIDX3_6]], align 1
-; CHECK-NEXT:    [[XOR12_6:%.*]] = xor i8 [[TMP13]], [[TMP12]]
-; CHECK-NEXT:    [[AND13_6:%.*]] = and i8 [[XOR12_6]], [[AND13_5]]
 ; CHECK-NEXT:    [[ARRAYIDX_7:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 7
-; CHECK-NEXT:    [[TMP14:%.*]] = load i8, i8* [[ARRAYIDX_7]], align 1
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[ARRAYIDX]] to <8 x i8>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i8>, <8 x i8>* [[TMP0]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_7:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 7
-; CHECK-NEXT:    [[TMP15:%.*]] = load i8, i8* [[ARRAYIDX3_7]], align 1
-; CHECK-NEXT:    [[XOR12_7:%.*]] = xor i8 [[TMP15]], [[TMP14]]
-; CHECK-NEXT:    [[AND13_7:%.*]] = and i8 [[XOR12_7]], [[AND13_6]]
-; CHECK-NEXT:    ret i8 [[AND13_7]]
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i8* [[ARRAYIDX3]] to <8 x i8>*
+; CHECK-NEXT:    [[TMP3:%.*]] = load <8 x i8>, <8 x i8>* [[TMP2]], align 1
+; CHECK-NEXT:    [[TMP4:%.*]] = xor <8 x i8> [[TMP3]], [[TMP1]]
+; CHECK-NEXT:    [[TMP5:%.*]] = call i8 @llvm.vector.reduce.and.v8i8(<8 x i8> [[TMP4]])
+; CHECK-NEXT:    [[OP_EXTRA:%.*]] = and i8 [[TMP5]], 1
+; CHECK-NEXT:    ret i8 [[OP_EXTRA]]
 ;
 entry:
   %arrayidx = getelementptr inbounds %struct.buf, %struct.buf* %a, i64 0, i32 0, i64 0
   %0 = load i8, i8* %arrayidx, align 1
   %arrayidx3 = getelementptr inbounds %struct.buf, %struct.buf* %b, i64 0, i32 0, i64 0
   %1 = load i8, i8* %arrayidx3, align 1
   %xor12 = xor i8 %1, %0
   %and13 = and i8 %xor12, 1

llvm/test/Transforms/SLPVectorizer/AArch64/slp-or-reduction.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -S -mtriple=aarch64 -slp-vectorizer | FileCheck %s
 
 %struct.buf = type { [8 x i8] }
 
 define i8 @reduce_or(%struct.buf* %a, %struct.buf* %b) {
 ; CHECK-LABEL: @reduce_or(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_BUF:%.*]], %struct.buf* [[A:%.*]], i64 0, i32 0, i64 0
-; CHECK-NEXT:    [[TMP0:%.*]] = load i8, i8* [[ARRAYIDX]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B:%.*]], i64 0, i32 0, i64 0
-; CHECK-NEXT:    [[TMP1:%.*]] = load i8, i8* [[ARRAYIDX3]], align 1
-; CHECK-NEXT:    [[XOR12:%.*]] = xor i8 [[TMP1]], [[TMP0]]
 ; CHECK-NEXT:    [[ARRAYIDX_1:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 1
-; CHECK-NEXT:    [[TMP2:%.*]] = load i8, i8* [[ARRAYIDX_1]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_1:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 1
-; CHECK-NEXT:    [[TMP3:%.*]] = load i8, i8* [[ARRAYIDX3_1]], align 1
-; CHECK-NEXT:    [[XOR12_1:%.*]] = xor i8 [[TMP3]], [[TMP2]]
-; CHECK-NEXT:    [[OR13_1:%.*]] = or i8 [[XOR12_1]], [[XOR12]]
 ; CHECK-NEXT:    [[ARRAYIDX_2:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 2
-; CHECK-NEXT:    [[TMP4:%.*]] = load i8, i8* [[ARRAYIDX_2]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_2:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 2
-; CHECK-NEXT:    [[TMP5:%.*]] = load i8, i8* [[ARRAYIDX3_2]], align 1
-; CHECK-NEXT:    [[XOR12_2:%.*]] = xor i8 [[TMP5]], [[TMP4]]
-; CHECK-NEXT:    [[OR13_2:%.*]] = or i8 [[XOR12_2]], [[OR13_1]]
 ; CHECK-NEXT:    [[ARRAYIDX_3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 3
-; CHECK-NEXT:    [[TMP6:%.*]] = load i8, i8* [[ARRAYIDX_3]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 3
-; CHECK-NEXT:    [[TMP7:%.*]] = load i8, i8* [[ARRAYIDX3_3]], align 1
-; CHECK-NEXT:    [[XOR12_3:%.*]] = xor i8 [[TMP7]], [[TMP6]]
-; CHECK-NEXT:    [[OR13_3:%.*]] = or i8 [[XOR12_3]], [[OR13_2]]
 ; CHECK-NEXT:    [[ARRAYIDX_4:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 4
-; CHECK-NEXT:    [[TMP8:%.*]] = load i8, i8* [[ARRAYIDX_4]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_4:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 4
-; CHECK-NEXT:    [[TMP9:%.*]] = load i8, i8* [[ARRAYIDX3_4]], align 1
-; CHECK-NEXT:    [[XOR12_4:%.*]] = xor i8 [[TMP9]], [[TMP8]]
-; CHECK-NEXT:    [[OR13_4:%.*]] = or i8 [[XOR12_4]], [[OR13_3]]
 ; CHECK-NEXT:    [[ARRAYIDX_5:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 5
-; CHECK-NEXT:    [[TMP10:%.*]] = load i8, i8* [[ARRAYIDX_5]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_5:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 5
-; CHECK-NEXT:    [[TMP11:%.*]] = load i8, i8* [[ARRAYIDX3_5]], align 1
-; CHECK-NEXT:    [[XOR12_5:%.*]] = xor i8 [[TMP11]], [[TMP10]]
-; CHECK-NEXT:    [[OR13_5:%.*]] = or i8 [[XOR12_5]], [[OR13_4]]
 ; CHECK-NEXT:    [[ARRAYIDX_6:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 6
-; CHECK-NEXT:    [[TMP12:%.*]] = load i8, i8* [[ARRAYIDX_6]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_6:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 6
-; CHECK-NEXT:    [[TMP13:%.*]] = load i8, i8* [[ARRAYIDX3_6]], align 1
-; CHECK-NEXT:    [[XOR12_6:%.*]] = xor i8 [[TMP13]], [[TMP12]]
-; CHECK-NEXT:    [[OR13_6:%.*]] = or i8 [[XOR12_6]], [[OR13_5]]
 ; CHECK-NEXT:    [[ARRAYIDX_7:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 7
-; CHECK-NEXT:    [[TMP14:%.*]] = load i8, i8* [[ARRAYIDX_7]], align 1
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[ARRAYIDX]] to <8 x i8>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i8>, <8 x i8>* [[TMP0]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_7:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 7
-; CHECK-NEXT:    [[TMP15:%.*]] = load i8, i8* [[ARRAYIDX3_7]], align 1
-; CHECK-NEXT:    [[XOR12_7:%.*]] = xor i8 [[TMP15]], [[TMP14]]
-; CHECK-NEXT:    [[OR13_7:%.*]] = or i8 [[XOR12_7]], [[OR13_6]]
-; CHECK-NEXT:    ret i8 [[OR13_7]]
-;
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i8* [[ARRAYIDX3]] to <8 x i8>*
+; CHECK-NEXT:    [[TMP3:%.*]] = load <8 x i8>, <8 x i8>* [[TMP2]], align 1
+; CHECK-NEXT:    [[TMP4:%.*]] = xor <8 x i8> [[TMP3]], [[TMP1]]
+; CHECK-NEXT:    [[TMP5:%.*]] = call i8 @llvm.vector.reduce.or.v8i8(<8 x i8> [[TMP4]])
+; CHECK-NEXT:    ret i8 [[TMP5]]
+
 entry:
   %arrayidx = getelementptr inbounds %struct.buf, %struct.buf* %a, i64 0, i32 0, i64 0
   %0 = load i8, i8* %arrayidx, align 1
   %arrayidx3 = getelementptr inbounds %struct.buf, %struct.buf* %b, i64 0, i32 0, i64 0
   %1 = load i8, i8* %arrayidx3, align 1
   %xor12 = xor i8 %1, %0
   %arrayidx.1 = getelementptr inbounds %struct.buf, %struct.buf* %a, i64 0, i32 0, i64 1
   %2 = load i8, i8* %arrayidx.1, align 1

llvm/test/Transforms/SLPVectorizer/AArch64/slp-xor-reduction.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -S -mtriple=aarch64 -slp-vectorizer | FileCheck %s
 
 %struct.buf = type { [8 x i8] }
 
 define i8 @reduce_xor(%struct.buf* %a, %struct.buf* %b) {
 ; CHECK-LABEL: @reduce_xor(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [[STRUCT_BUF:%.*]], %struct.buf* [[A:%.*]], i64 0, i32 0, i64 0
-; CHECK-NEXT:    [[TMP0:%.*]] = load i8, i8* [[ARRAYIDX]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B:%.*]], i64 0, i32 0, i64 0
-; CHECK-NEXT:    [[TMP1:%.*]] = load i8, i8* [[ARRAYIDX3]], align 1
-; CHECK-NEXT:    [[AND12:%.*]] = and i8 [[TMP1]], [[TMP0]]
 ; CHECK-NEXT:    [[ARRAYIDX_1:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 1
-; CHECK-NEXT:    [[TMP2:%.*]] = load i8, i8* [[ARRAYIDX_1]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_1:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 1
-; CHECK-NEXT:    [[TMP3:%.*]] = load i8, i8* [[ARRAYIDX3_1]], align 1
-; CHECK-NEXT:    [[AND12_1:%.*]] = and i8 [[TMP3]], [[TMP2]]
-; CHECK-NEXT:    [[TMP4:%.*]] = xor i8 [[AND12]], [[AND12_1]]
 ; CHECK-NEXT:    [[ARRAYIDX_2:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 2
-; CHECK-NEXT:    [[TMP5:%.*]] = load i8, i8* [[ARRAYIDX_2]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_2:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 2
-; CHECK-NEXT:    [[TMP6:%.*]] = load i8, i8* [[ARRAYIDX3_2]], align 1
-; CHECK-NEXT:    [[AND12_2:%.*]] = and i8 [[TMP6]], [[TMP5]]
-; CHECK-NEXT:    [[TMP7:%.*]] = xor i8 [[TMP4]], [[AND12_2]]
 ; CHECK-NEXT:    [[ARRAYIDX_3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 3
-; CHECK-NEXT:    [[TMP8:%.*]] = load i8, i8* [[ARRAYIDX_3]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_3:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 3
-; CHECK-NEXT:    [[TMP9:%.*]] = load i8, i8* [[ARRAYIDX3_3]], align 1
-; CHECK-NEXT:    [[AND12_3:%.*]] = and i8 [[TMP9]], [[TMP8]]
-; CHECK-NEXT:    [[TMP10:%.*]] = xor i8 [[TMP7]], [[AND12_3]]
 ; CHECK-NEXT:    [[ARRAYIDX_4:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 4
-; CHECK-NEXT:    [[TMP11:%.*]] = load i8, i8* [[ARRAYIDX_4]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_4:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 4
-; CHECK-NEXT:    [[TMP12:%.*]] = load i8, i8* [[ARRAYIDX3_4]], align 1
-; CHECK-NEXT:    [[AND12_4:%.*]] = and i8 [[TMP12]], [[TMP11]]
-; CHECK-NEXT:    [[TMP13:%.*]] = xor i8 [[TMP10]], [[AND12_4]]
 ; CHECK-NEXT:    [[ARRAYIDX_5:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 5
-; CHECK-NEXT:    [[TMP14:%.*]] = load i8, i8* [[ARRAYIDX_5]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_5:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 5
-; CHECK-NEXT:    [[TMP15:%.*]] = load i8, i8* [[ARRAYIDX3_5]], align 1
-; CHECK-NEXT:    [[AND12_5:%.*]] = and i8 [[TMP15]], [[TMP14]]
-; CHECK-NEXT:    [[TMP16:%.*]] = xor i8 [[TMP13]], [[AND12_5]]
 ; CHECK-NEXT:    [[ARRAYIDX_6:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 6
-; CHECK-NEXT:    [[TMP17:%.*]] = load i8, i8* [[ARRAYIDX_6]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_6:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 6
-; CHECK-NEXT:    [[TMP18:%.*]] = load i8, i8* [[ARRAYIDX3_6]], align 1
-; CHECK-NEXT:    [[AND12_6:%.*]] = and i8 [[TMP18]], [[TMP17]]
-; CHECK-NEXT:    [[TMP19:%.*]] = xor i8 [[TMP16]], [[AND12_6]]
 ; CHECK-NEXT:    [[ARRAYIDX_7:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[A]], i64 0, i32 0, i64 7
-; CHECK-NEXT:    [[TMP20:%.*]] = load i8, i8* [[ARRAYIDX_7]], align 1
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[ARRAYIDX]] to <8 x i8>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <8 x i8>, <8 x i8>* [[TMP0]], align 1
 ; CHECK-NEXT:    [[ARRAYIDX3_7:%.*]] = getelementptr inbounds [[STRUCT_BUF]], %struct.buf* [[B]], i64 0, i32 0, i64 7
-; CHECK-NEXT:    [[TMP21:%.*]] = load i8, i8* [[ARRAYIDX3_7]], align 1
-; CHECK-NEXT:    [[AND12_7:%.*]] = and i8 [[TMP21]], [[TMP20]]
-; CHECK-NEXT:    [[TMP22:%.*]] = xor i8 [[TMP19]], [[AND12_7]]
-; CHECK-NEXT:    [[XOR13_7:%.*]] = xor i8 [[TMP22]], 1
-; CHECK-NEXT:    ret i8 [[XOR13_7]]
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i8* [[ARRAYIDX3]] to <8 x i8>*
+; CHECK-NEXT:    [[TMP3:%.*]] = load <8 x i8>, <8 x i8>* [[TMP2]], align 1
+; CHECK-NEXT:    [[TMP4:%.*]] = and <8 x i8> [[TMP3]], [[TMP1]]
+; CHECK-NEXT:    [[TMP5:%.*]] = call i8 @llvm.vector.reduce.xor.v8i8(<8 x i8> [[TMP4]])
+; CHECK-NEXT:    [[OP_EXTRA:%.*]] = xor i8 [[TMP5]], 1
+; CHECK-NEXT:    ret i8 [[OP_EXTRA]]
 ;
 entry:
   %arrayidx = getelementptr inbounds %struct.buf, %struct.buf* %a, i64 0, i32 0, i64 0
   %0 = load i8, i8* %arrayidx, align 1
   %arrayidx3 = getelementptr inbounds %struct.buf, %struct.buf* %b, i64 0, i32 0, i64 0
   %1 = load i8, i8* %arrayidx3, align 1
   %and12 = and i8 %1, %0
   %arrayidx.1 = getelementptr inbounds %struct.buf, %struct.buf* %a, i64 0, i32 0, i64 1