[InstCombine] Enable fold select into operand for FAdd, FMul, FSub and FDiv.

[huihuiz]

Summary

For FAdd, FMul, FSub and FDiv, fold select into one of the operands to enable
further optimizations, i.e., floating-point reduction detection.

Turn code:

%C = fadd %A, %B
%D = select %cond, %C, %A

into:

%C = select %cond, %B, -0.000000e+00
%D = fadd %A, %C

Alive2 verification (with --disable-undef-input), timed out otherwise.
FAdd - https://alive2.llvm.org/ce/z/eUxN4Y
FMul - https://alive2.llvm.org/ce/z/5SWZz4
FSub - https://alive2.llvm.org/ce/z/Dhj8dU
FDiv - https://alive2.llvm.org/ce/z/Yj_NA2

[huihuiz]

Take test.ll attached.

Run: opt -polly-process-unprofitable -polly-remarks-minimal -polly-use-llvm-names -polly-codegen-verify -analyze -polly-scops test.ll
Then you will see reduction is not detected.
MustWriteAccess := [Reduction Type: NONE] [Scalar: 0]

• -> { Stmt_for_body[i0] -> MemRef_sum_014_reg2mem[0] };

After enable fold select into operand for FAdd, then you will see reduction is detected.
run: opt -S -instcombine test.ll -o test2.ll (run with this patch)
opt -polly-process-unprofitable -polly-remarks-minimal -polly-use-llvm-names -polly-codegen-verify -analyze -polly-scops test2.ll

MustWriteAccess := [Reduction Type: +] [Scalar: 0]

• -> { Stmt_for_body[i0] -> MemRef_sum_014_reg2mem[0] };

target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-gnu"

define float @test(i32 %n, float* noalias nocapture readonly %a) {
entry:
  %sum.014.reg2mem = alloca float, align 4
  %sum.0.lcssa.reg2mem = alloca float, align 4
  br label %entry.split15

entry.split15:                                    ; preds = %entry
  br label %entry.split

entry.split:                                      ; preds = %entry.split15
  %cmp12 = icmp sgt i32 %n, 0
  store float 0.000000e+00, float* %sum.0.lcssa.reg2mem, align 4
  br i1 %cmp12, label %for.body.preheader, label %for.end

for.body.preheader:                               ; preds = %entry.split
  %wide.trip.count = zext i32 %n to i64
  store float 0.000000e+00, float* %sum.014.reg2mem, align 4
  br label %for.body

for.body:                                         ; preds = %for.body.preheader, %for.body
  %indvars.iv = phi i64 [ 0, %for.body.preheader ], [ %indvars.iv.next, %for.body ]
  %sum.014.reload = load float, float* %sum.014.reg2mem, align 4
  %arrayidx = getelementptr inbounds float, float* %a, i64 %indvars.iv
  %0 = load float, float* %arrayidx, align 4
  %cmp1 = fcmp fast ogt float %0, 0.000000e+00
  %add = fadd fast float %0, %sum.014.reload
  %sum.1 = select i1 %cmp1, float %add, float %sum.014.reload
  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
  %exitcond.not = icmp eq i64 %indvars.iv.next, %wide.trip.count
  store float %sum.1, float* %sum.014.reg2mem, align 4
  br i1 %exitcond.not, label %for.end.loopexit, label %for.body

for.end.loopexit:                                 ; preds = %for.body
  %1 = load float, float* %sum.014.reg2mem, align 4
  store float %1, float* %sum.0.lcssa.reg2mem, align 4
  br label %for.end

for.end:                                          ; preds = %for.end.loopexit, %entry.split
  %sum.0.lcssa.reload = load float, float* %sum.0.lcssa.reg2mem, align 4
  ret float %sum.0.lcssa.reload
}

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

257

Why exclude fdiv?

llvm/test/Transforms/InstCombine/select-binop-cmp.ll

307

Please pre-commit the baseline tests. Some of these tests should include fast-math-flags on the FP binop, so we can verify that FMF propagates as expected.

330

Why is the compare constant (0.0 or -0.0) relevant for this fold?

The true/false operands should be swapped so we have coverage for the pattern that replaces the true value with a constant. Similarly for fmul, there should be two tests.

[spatel]

llvm/test/Transforms/InstCombine/select-binop-cmp.ll

330

A better question might be - why is there an fcmp in any of these tests? That isn't part of the minimal pattern is it?

[huihuiz]

Thanks Sanjay for the comments, I will update unit test as suggested.

I have some concern when I checked with alive2, for fadd https://alive2.llvm.org/ce/z/UjAMM_ alive2 complaints mis-matched outputs.
For integer add this folding seems correct https://alive2.llvm.org/ce/z/oE4UQJ

Let me know if such transformation would be illegal for floating point type as alive2 pointed out, or there is anything I missed before I go fixing the unit test ?

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

257

I checked with alive2, looks like adding fdiv, sdiv and udiv will trigger undefined behavior
https://alive2.llvm.org/ce/z/KvFYev

[spatel]

In D113442#3119977, @huihuiz wrote:

I have some concern when I checked with alive2, for fadd https://alive2.llvm.org/ce/z/UjAMM_ alive2 complaints mis-matched outputs.

For fadd, we need to use -0.0 as the binop identity constant. I'm not getting the online instance of Alive2 to verify this without timing out, but I think this should work given enough time:
https://alive2.llvm.org/ce/z/TtGDNR

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

257

I think the integer ops are not safe because we can trigger immediate UB that might have been avoided in the original code, but fdiv should be fine - it doesn't have any different UB characteristics vs. fadd/fmul/fsub in the default FP environment.
https://alive2.llvm.org/ce/z/Yj_NA2
(This is timing out on the online version when allowing undef/poison, so please double-check on a local machine.)

[huihuiz]

Addressed review comments.
Pre-commit baseline test test/Transforms/InstCombine/select-binop-foldable-floating-point.ll

[huihuiz]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

257

Thanks Sanjay for the explanation. Include 'fdiv' to allow fold on the divisor amount.

I did alive-tv run on my local machine with time out increased to
./alive-tv src.ll tgt.ll -smt-to 100000000

But alive did not converge after 1.5 hour. Doing an overnight run now, will update result tomorrow morning.

llvm/test/Transforms/InstCombine/select-binop-cmp.ll

330

Removing fcmp instruction. The minimal pattern to fold select into binop operand does not require a fcmp. I am moving this into a separate test. The original test select-binop-cmp.ll checks fcmp ignores the sign of 0.0 (for example test @select_fadd_fcmp), so keep the original test unchanged for now.

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

257

Hmm...I think it's a good sign if it did not find a failure after 1.5 hours, but that is a long time to spend on 2 instructions.
cc @nlopes @regehr @aqjune in case they see something wrong or room for improvement here:
https://alive2.llvm.org/ce/z/TtGDNR

[huihuiz]

Update on overnight run for fdiv

My run below did not return anything or capture any failure after 12 hours. 1000000000 is probably the maximum time out we can set to alive.
But I do agree that for two instructions, alive is taking too long to converge.

Definitely point us out if there is anything wrong ?

./alive-tv src.ll -smt-to 1000000000

----------------------------------------
define half @select_fdiv(i1 %cond, half %A, half %B) {
%0:
  %C = fdiv half %A, %B
  %D = select i1 %cond, half %C, half %A
  ret half %D
}
=>
define half @select_fdiv(i1 %cond, half %A, half %B) {
%0:
  %C = select i1 %cond, half %B, half 15360
  %D = fdiv half %A, %C
  ret half %D
}

[huihuiz]

Rebased, and a gentle ping ?

Let me know if there are concerns with the current implementation ?

[spatel]

LGTM.

You wrote that fdiv ran overnight without completing. I wonder if that opcode is slower than the others. For example, did fadd/fsub also timeout?

[huihuiz]

Thanks Sanjay for the review!
I did another local run for fadd, with "--disable-undef-input" it finish within a minute.
When removing "--disable-undef-input", it's taking about an hour now, still not finished.

Probably, the problem is the floating-point type. We hit some limitations of alive2 I assume.

[nlopes]

In D113442#3146921, @huihuiz wrote:

Thanks Sanjay for the review!
I did another local run for fadd, with "--disable-undef-input" it finish within a minute.
When removing "--disable-undef-input", it's taking about an hour now, still not finished.

Probably, the problem is the floating-point type. We hit some limitations of alive2 I assume.

Reasoning about floats is already quite expensive, and mixing that with undefs is really hard.
The fact it doesn't find any bug within 1 hour is a good sign though. Right now there's nothing better we can do. We have plans to improve this in the longer term.

[goncharov]

Just FYI: I found that this change might affect of Ptrdist-ks benchmark from MultiSource suite on AMD Rome cores up to 30%. Don't have much details but leaving it here in case anyone else is affected.

[LuoYuanke]

This patch cause regression (https://godbolt.org/z/7WYboTe16) on x86 avx512 target, because avx512 has mask instruction which can match select instruction in ISel pattern match. Move select before fadd cause the pattern match fails. I don't quite understand for the optimization opportunity for this transformation. Could you elaborate an example for it?

This patch has side effect on AVX512 ISel. Another example is https://godbolt.org/z/xdr8xs4cb.

[xbolva00]

This needs backend undo transformation to be implemented first.

Plesse revert @huihuiz

[xbolva00]

In D113442#3281729, @LuoYuanke wrote:

This patch cause regression (https://godbolt.org/z/7WYboTe16) on x86 avx512 target, because avx512 has mask instruction which can match select instruction in ISel pattern match. Move select before fadd cause the pattern match fails. I don't quite understand for the optimization opportunity for this transformation. Could you elaborate an example for it?

This patch has side effect on AVX512 ISel. Another example is https://godbolt.org/z/xdr8xs4cb.

Yeah, this patch was commited without checking how this transformation affects x86 codegen. :/ needs to be reverted before LLVM 14.

cc @spatel

[spatel]

In D113442#3281741, @xbolva00 wrote:

In D113442#3281729, @LuoYuanke wrote:

This patch cause regression (https://godbolt.org/z/7WYboTe16) on x86 avx512 target, because avx512 has mask instruction which can match select instruction in ISel pattern match. Move select before fadd cause the pattern match fails. I don't quite understand for the optimization opportunity for this transformation. Could you elaborate an example for it?

This patch has side effect on AVX512 ISel. Another example is https://godbolt.org/z/xdr8xs4cb.

Yeah, this patch was commited without checking how this transformation affects x86 codegen. :/ needs to be reverted before LLVM 14.

cc @spatel

Note that this patch makes IR more consistent (FP ops are treated the same as integer ops). So is the integer codegen also not optimal?
https://godbolt.org/z/jd3q6Yq55

If there's still time for clang 14, can we fix x86 codegen rather than revert? There was already a similar bug filed for compares:
https://github.com/llvm/llvm-project/issues/51842

[LuoYuanke]

Note that this patch makes IR more consistent (FP ops are treated the same as integer ops). So is the integer codegen also not optimal?
https://godbolt.org/z/jd3q6Yq55

I think the integer is also NOT optimal with AVX512 target. See https://godbolt.org/z/Wahef3b3v. For _mm_mask_add_epi64() and _mm_mask_sub_epi64(), it eventually generate extra instructions. But for integer mul, or, and, xor and shl in the test case the InstCombinePass doesn't transform it, so it still generate good code.

I still don't understand what's the benefit to move select instruction before the binary operation (add, sub, mul ...). Can someone indicate why it is profitable? If it is profitable for general case, I think we can call TTI interface to ask backend if the transform is profitable and then make the decision to transform. We may add below interfaces in TTI.

+
+  bool isLegalMaskedFAdd(Type *DataTye);
+  bool isLegalMaskedFSub(Type *DataTye);
+  bool isLegalMaskedFMul(Type *DataTye);
+  bool isLegalMaskedFDiv(Type *DataTye);
+

[craig.topper]

In D113442#3282412, @LuoYuanke wrote:

Note that this patch makes IR more consistent (FP ops are treated the same as integer ops). So is the integer codegen also not optimal?
https://godbolt.org/z/jd3q6Yq55

I think the integer is also NOT optimal with AVX512 target. See https://godbolt.org/z/Wahef3b3v. For _mm_mask_add_epi64() and _mm_mask_sub_epi64(), it eventually generate extra instructions. But for integer mul, or, and, xor and shl in the test case the InstCombinePass doesn't transform it, so it still generate good code.

I still don't understand what's the benefit to move select instruction before the binary operation (add, sub, mul ...). Can someone indicate why it is profitable? If it is profitable for general case, I think we can call TTI interface to ask backend if the transform is profitable and then make the decision to transform. We may add below interfaces in TTI.

+
+  bool isLegalMaskedFAdd(Type *DataTye);
+  bool isLegalMaskedFSub(Type *DataTye);
+  bool isLegalMaskedFMul(Type *DataTye);
+  bool isLegalMaskedFDiv(Type *DataTye);
+

I believe at least one reason for the int combine was because it reduces the number of uses of the input.

ARM and RISCV both reverse it in the backend for scalars. Though they should probably be using FREEZE to do it.

X86 could probably reverse it for vectors.

[dmgreen]

For vectors - Arm MVE has to reverse these transforms (as in https://reviews.llvm.org/rGd9af9c2c5a53c9ba6aa0255240a2a40e8bea27aa). It is simpler in general to match vselect cc, (add x, y), x) as a predicated-add, than it is with a folded select with an identity element. But we do manage to reverse that at the moment, and I've not seen any cases of it folding to something we couldn't convert back. It can be quite important for performance in places, for example where the vectorizer produces a predicated reduction (https://github.com/llvm/llvm-project/blob/main/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp#L4158, that is only enabled for Arm MVE at the moment).

SVE doesn't seem to attempt to convert add+select into predicate-add yet. https://godbolt.org/z/TPbE95h5x

[xbolva00]

Maybe worth to mention:
https://reviews.llvm.org/D90113

[spatel]

In D113442#3282819, @xbolva00 wrote:

Maybe worth to mention:
https://reviews.llvm.org/D90113

Thanks for the link. I posted a much more limited x86-only patch -- D118644 -- to deal with the regression noted here.

[huihuiz]

Thank you guys for looking into this!

One of the optimization enabled through this patch is reduction detection. The same applies to it's integer equivalent.

Take the example test.ll attached in the very first comment
run:
opt -S -instcombine test.ll -o test2.ll
opt -polly-process-unprofitable -polly-remarks-minimal -polly-use-llvm-names -polly-codegen-verify -analyze -polly-scops test2.ll

Then you will see reduction getting detected. Eventually allow loops to get vectorized.
MustWriteAccess := [Reduction Type: +] [Scalar: 0]

The folding in particular reduce the user of %sum.014.reload to 1. So that we don't need to over-complicate the data-flow analysis algorithm used by vectorizer. Also don't need to combine multiple reduction operators.

before

%cmp1 = fcmp fast ogt float %0, 0.000000e+00
%add = fadd fast float %0, %sum.014.reload
%sum.1 = select i1 %cmp1, float %add, float %sum.014.reload

after

%.inv = fcmp fast ole float %0, -0.000000e+00
%1 = select fast i1 %.inv, float -0.000000e+00, float %0
%sum.1 = fadd fast float %sum.014.reload, %1

[xbolva00]

In D113442#3206100, @goncharov wrote:

Just FYI: I found that this change might affect of Ptrdist-ks benchmark from MultiSource suite on AMD Rome cores up to 30%. Don't have much details but leaving it here in case anyone else is affected.

This is huge regression. Does @spatel ´s patch fix it?

Any performance measurements of this patch alone? Or phoronix sill surprise (good/bad?) us again

[RKSimon]

In D113442#3287665, @xbolva00 wrote:

In D113442#3206100, @goncharov wrote:

Just FYI: I found that this change might affect of Ptrdist-ks benchmark from MultiSource suite on AMD Rome cores up to 30%. Don't have much details but leaving it here in case anyone else is affected.

This is huge regression. Does @spatel ´s patch fix it?

D118644 only handles vectors and AVX512 targets so this regression wouldn't be fixed

[spatel]

Following-up:
https://github.com/llvm/llvm-project/issues/53866 lists the commits that are in main/branch to avoid the AVX512 FP regressions.

Further work to invert this transform in the backend is ongoing. For example:
D119654

This has to be done in relatively small steps to avoid regressions on x86 - it's not clear without looking at asm if the transformed code will be better (may depend on operation, data type, and subtarget features).

llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp

 ///   %D = or %A, %C
 ///
 /// Assuming that the specified instruction is an operand to the select, return
 /// a bitmask indicating which operands of this instruction are foldable if they
 /// equal the other incoming value of the select.
 static unsigned getSelectFoldableOperands(BinaryOperator *I) {
   switch (I->getOpcode()) {
   case Instruction::Add:
+  case Instruction::FAdd:
   case Instruction::Mul:
+  case Instruction::FMul:
   case Instruction::And:
   case Instruction::Or:
   case Instruction::Xor:
     return 3;              // Can fold through either operand.
   case Instruction::Sub:   // Can only fold on the amount subtracted.
+  case Instruction::FSub:
   case Instruction::Shl:   // Can only fold on the shift amount.
   case Instruction::LShr:
   case Instruction::AShr:
     return 1;
   default:
     return 0;              // Cannot fold
   }
 }

llvm/test/Transforms/InstCombine/select-binop-cmp.ll

 ; CHECK-NEXT:    ret float [[C]]
 ;
   %A = fcmp oeq float %x, 0.0
   %B = fadd float %x, 6.0
   %C = select i1 %A, float %B, float %y
   ret float %C
 }
 
+define float @select_fadd_fcmp_7_fold_select_into_op(float %x, float %y, float %z) {
+; CHECK-LABEL: @select_fadd_fcmp_7_fold_select_into_op(
+; CHECK-NEXT:    [[TMP1:%.*]] = fcmp ogt float [[X:%.*]], -0.000000e+00
+; CHECK-NEXT:    [[B1:%.*]] = select i1 [[TMP1]], float [[X]], float -0.000000e+00
+; CHECK-NEXT:    [[C:%.*]] = fadd float [[B1]], [[Z:%.*]]
+; CHECK-NEXT:    ret float [[C]]
+;
+  %A = fcmp ogt float %x, -0.0
+  %B = fadd float %x, %z
+  %C = select i1 %A, float %B, float %z
+  ret float %C
+}
+
+; This is logically equivalent to the previous test - fcmp ignores the sign of 0.0.
+define float @select_fadd_fcmp_7_fold_select_into_op_poszero(float %x, float %y, float %z) {
+; CHECK-LABEL: @select_fadd_fcmp_7_fold_select_into_op_poszero(
+; CHECK-NEXT:    [[TMP1:%.*]] = fcmp ogt float [[X:%.*]], -0.000000e+00
+; CHECK-NEXT:    [[B1:%.*]] = select i1 [[TMP1]], float [[X]], float -0.000000e+00
+; CHECK-NEXT:    [[C:%.*]] = fadd float [[B1]], [[Z:%.*]]
+; CHECK-NEXT:    ret float [[C]]
+;
+  %A = fcmp ogt float %x, 0.0
+  %B = fadd float %x, %z
+  %C = select i1 %A, float %B, float %z
+  ret float %C
+}
+
 define float @select_fmul_fcmp(float %x, float %y, float %z) {
 ; CHECK-LABEL: @select_fmul_fcmp(
 ; CHECK-NEXT:    [[A:%.*]] = fcmp oeq float [[X:%.*]], 1.000000e+00
 ; CHECK-NEXT:    [[C:%.*]] = select i1 [[A]], float [[Z:%.*]], float [[Y:%.*]]
 ; CHECK-NEXT:    ret float [[C]]
 ;
   %A = fcmp oeq float %x, 1.0
   %B = fmul nsz float %x, %z
   %C = select i1 %A, float %B, float %y
   ret float %C
 }
 
+define float @select_fmul_fcmp_fold_select_into_op(float %x, float %y, float %z) {
+; CHECK-LABEL: @select_fmul_fcmp_fold_select_into_op(
+; CHECK-NEXT:    [[A:%.*]] = fcmp ogt float [[X:%.*]], 2.000000e+00
+; CHECK-NEXT:    [[B:%.*]] = select i1 [[A]], float [[X]], float 1.000000e+00
+; CHECK-NEXT:    [[C:%.*]] = fmul float [[B]], [[Z:%.*]]
+; CHECK-NEXT:    ret float [[C]]
+;
+  %A = fcmp ogt float %x, 2.0
+  %B = fmul float %x, %z
+  %C = select i1 %A, float %B, float %z
+  ret float %C
+}
+
 define float @select_fsub_fcmp(float %x, float %y, float %z) {
 ; CHECK-LABEL: @select_fsub_fcmp(
 ; CHECK-NEXT:    [[A:%.*]] = fcmp oeq float [[X:%.*]], 0.000000e+00
 ; CHECK-NEXT:    [[C:%.*]] = select i1 [[A]], float [[Z:%.*]], float [[Y:%.*]]
 ; CHECK-NEXT:    ret float [[C]]
 ;
   %A = fcmp oeq float %x, 0.0
   %B = fsub nsz float %z, %x

 ; CHECK-NEXT:    ret float [[C]]
 ;
   %A = fcmp oeq float %x, -0.0
   %B = fsub nsz float %z, %x
   %C = select i1 %A, float %B, float %y
   ret float %C
 }
 
+define float @select_fsub_fcmp_fold_select_into_op(float %x, float %y) {
+; CHECK-LABEL: @select_fsub_fcmp_fold_select_into_op(
+; CHECK-NEXT:    [[C:%.*]] = fcmp one float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[SUB:%.*]] = select i1 [[C]], float [[X]], float 0.000000e+00
+; CHECK-NEXT:    [[S:%.*]] = fsub float [[Y:%.*]], [[SUB]]
+; CHECK-NEXT:    ret float [[S]]
+;
+  %c = fcmp one float %x, 0.0
+  %sub = fsub float %y, %x
+  %s = select i1 %c, float %sub, float %y
+  ret float %s
+}
+
+; This is logically equivalent to the previous test - fcmp ignores the sign of 0.0.
+define float @select_fsub_fcmp_fold_select_into_op_negzero(float %x, float %y) {
+; CHECK-LABEL: @select_fsub_fcmp_fold_select_into_op_negzero(
+; CHECK-NEXT:    [[C:%.*]] = fcmp one float [[X:%.*]], 0.000000e+00
+; CHECK-NEXT:    [[SUB:%.*]] = select i1 [[C]], float [[X]], float 0.000000e+00
+; CHECK-NEXT:    [[S:%.*]] = fsub float [[Y:%.*]], [[SUB]]
+; CHECK-NEXT:    ret float [[S]]
+;
+  %c = fcmp one float %x, -0.0
+  %sub = fsub float %y, %x
+  %s = select i1 %c, float %sub, float %y
+  ret float %s
+}
+
 define float @select_fdiv_fcmp(float %x, float %y, float %z) {
 ; CHECK-LABEL: @select_fdiv_fcmp(
 ; CHECK-NEXT:    [[A:%.*]] = fcmp oeq float [[X:%.*]], 1.000000e+00
 ; CHECK-NEXT:    [[C:%.*]] = select i1 [[A]], float [[Z:%.*]], float [[Y:%.*]]
 ; CHECK-NEXT:    ret float [[C]]
 ;
   %A = fcmp oeq float %x, 1.0
   %B = fdiv nsz float %z, %x

llvm/test/Transforms/LoopVectorize/ARM/mve-selectandorcost.ll

 ; CHECK-NEXT:    [[TMP3:%.*]] = fcmp fast oeq <4 x float> [[WIDE_LOAD6]], zeroinitializer
 ; CHECK-NEXT:    [[TMP4:%.*]] = select <4 x i1> [[TMP2]], <4 x i1> [[TMP3]], <4 x i1> zeroinitializer
 ; CHECK-NEXT:    [[TMP5:%.*]] = call fast <4 x float> @llvm.fabs.v4f32(<4 x float> [[WIDE_LOAD]])
 ; CHECK-NEXT:    [[TMP6:%.*]] = call fast <4 x float> @llvm.fabs.v4f32(<4 x float> [[WIDE_LOAD6]])
 ; CHECK-NEXT:    [[TMP7:%.*]] = fadd fast <4 x float> [[TMP6]], [[TMP5]]
 ; CHECK-NEXT:    [[TMP8:%.*]] = fsub fast <4 x float> [[WIDE_LOAD]], [[WIDE_LOAD6]]
 ; CHECK-NEXT:    [[TMP9:%.*]] = call fast <4 x float> @llvm.fabs.v4f32(<4 x float> [[TMP8]])
 ; CHECK-NEXT:    [[TMP10:%.*]] = fdiv fast <4 x float> [[TMP9]], [[TMP7]]
-; CHECK-NEXT:    [[TMP11:%.*]] = fadd fast <4 x float> [[TMP10]], [[VEC_PHI]]
-; CHECK-NEXT:    [[PREDPHI]] = select <4 x i1> [[TMP4]], <4 x float> [[VEC_PHI]], <4 x float> [[TMP11]]
+; CHECK-NEXT:    [[TMP11:%.*]] = select <4 x i1> [[TMP4]], <4 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, <4 x float> [[TMP10]]
+; CHECK-NEXT:    [[PREDPHI]] = fadd fast <4 x float> [[VEC_PHI]], [[TMP11]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 4
 ; CHECK-NEXT:    [[TMP12:%.*]] = icmp eq i32 [[INDEX_NEXT]], [[N_VEC]]
 ; CHECK-NEXT:    br i1 [[TMP12]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    [[TMP13:%.*]] = call fast float @llvm.vector.reduce.fadd.v4f32(float -0.000000e+00, <4 x float> [[PREDPHI]])
 ; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i32 [[N_VEC]], [[BLOCKSIZE]]
 ; CHECK-NEXT:    br i1 [[CMP_N]], label [[WHILE_END]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:

llvm/test/Transforms/LoopVectorize/reduction-inloop-pred.ll

 ; CHECK-LABEL: @reduction_fadd(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE6:%.*]] ]
 ; CHECK-NEXT:    [[VEC_IND:%.*]] = phi <4 x i64> [ <i64 0, i64 1, i64 2, i64 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], [[PRED_LOAD_CONTINUE6]] ]
-; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi float [ 0.000000e+00, [[VECTOR_PH]] ], [ [[TMP45:%.*]], [[PRED_LOAD_CONTINUE6]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi float [ 0.000000e+00, [[VECTOR_PH]] ], [ [[TMP43:%.*]], [[PRED_LOAD_CONTINUE6]] ]
 ; CHECK-NEXT:    [[TMP0:%.*]] = icmp ult <4 x i64> [[VEC_IND]], <i64 257, i64 257, i64 257, i64 257>
 ; CHECK-NEXT:    [[TMP1:%.*]] = extractelement <4 x i1> [[TMP0]], i32 0
 ; CHECK-NEXT:    br i1 [[TMP1]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
 ; CHECK:       pred.load.if:
 ; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds float, float* [[A:%.*]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP3:%.*]] = load float, float* [[TMP2]], align 4
 ; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <4 x float> poison, float [[TMP3]], i32 0
 ; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds float, float* [[B:%.*]], i64 [[INDEX]]

 ; CHECK-NEXT:    [[TMP37:%.*]] = insertelement <4 x float> [[TMP29]], float [[TMP36]], i32 3
 ; CHECK-NEXT:    br label [[PRED_LOAD_CONTINUE6]]
 ; CHECK:       pred.load.continue6:
 ; CHECK-NEXT:    [[TMP38:%.*]] = phi <4 x float> [ [[TMP28]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP34]], [[PRED_LOAD_IF5]] ]
 ; CHECK-NEXT:    [[TMP39:%.*]] = phi <4 x float> [ [[TMP29]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP37]], [[PRED_LOAD_IF5]] ]
 ; CHECK-NEXT:    [[TMP40:%.*]] = select fast <4 x i1> [[TMP0]], <4 x float> [[TMP38]], <4 x float> zeroinitializer
 ; CHECK-NEXT:    [[TMP41:%.*]] = call fast float @llvm.vector.reduce.fadd.v4f32(float [[VEC_PHI]], <4 x float> [[TMP40]])
 ; CHECK-NEXT:    [[TMP42:%.*]] = select fast <4 x i1> [[TMP0]], <4 x float> [[TMP39]], <4 x float> zeroinitializer
-; CHECK-NEXT:    [[TMP43:%.*]] = call fast float @llvm.vector.reduce.fadd.v4f32(float [[TMP41]], <4 x float> [[TMP42]])
+; CHECK-NEXT:    [[TMP43]] = call fast float @llvm.vector.reduce.fadd.v4f32(float [[TMP41]], <4 x float> [[TMP42]])
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add i64 [[INDEX]], 4
 ; CHECK-NEXT:    [[VEC_IND_NEXT]] = add <4 x i64> [[VEC_IND]], <i64 4, i64 4, i64 4, i64 4>
 ; CHECK-NEXT:    [[TMP44:%.*]] = icmp eq i64 [[INDEX_NEXT]], 260
 ; CHECK-NEXT:    br i1 [[TMP44]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP20:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]

 ; CHECK-NEXT:    [[TMP6:%.*]] = fcmp ule <4 x float> [[WIDE_LOAD1]], <float 1.000000e+00, float 1.000000e+00, float 1.000000e+00, float 1.000000e+00>
 ; CHECK-NEXT:    [[TMP7:%.*]] = fcmp ogt <4 x float> [[WIDE_LOAD]], <float 2.000000e+00, float 2.000000e+00, float 2.000000e+00, float 2.000000e+00>
 ; CHECK-NEXT:    [[TMP8:%.*]] = and <4 x i1> [[TMP5]], [[TMP6]]
 ; CHECK-NEXT:    [[TMP9:%.*]] = and <4 x i1> [[TMP8]], [[TMP7]]
 ; CHECK-NEXT:    [[TMP10:%.*]] = xor <4 x i1> [[TMP7]], <i1 true, i1 true, i1 true, i1 true>
 ; CHECK-NEXT:    [[TMP11:%.*]] = and <4 x i1> [[TMP8]], [[TMP10]]
 ; CHECK-NEXT:    [[TMP12:%.*]] = xor <4 x i1> [[TMP5]], <i1 true, i1 true, i1 true, i1 true>
 ; CHECK-NEXT:    [[PREDPHI_V:%.*]] = select <4 x i1> [[TMP9]], <4 x float> [[WIDE_LOAD1]], <4 x float> [[WIDE_LOAD]]
-; CHECK-NEXT:    [[PREDPHI:%.*]] = fadd fast <4 x float> [[VEC_PHI]], [[PREDPHI_V]]
 ; CHECK-NEXT:    [[TMP13:%.*]] = select <4 x i1> [[TMP12]], <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i1> [[TMP11]]
-; CHECK-NEXT:    [[PREDPHI3]] = select <4 x i1> [[TMP13]], <4 x float> [[VEC_PHI]], <4 x float> [[PREDPHI]]
+; CHECK-NEXT:    [[PREDPHI2:%.*]] = select <4 x i1> [[TMP13]], <4 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, <4 x float> [[PREDPHI_V]]
+; CHECK-NEXT:    [[PREDPHI3]] = fadd fast <4 x float> [[VEC_PHI]], [[PREDPHI2]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
 ; CHECK-NEXT:    [[TMP14:%.*]] = icmp eq i64 [[INDEX_NEXT]], 128
 ; CHECK-NEXT:    br i1 [[TMP14]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP28:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    [[TMP15:%.*]] = call fast float @llvm.vector.reduce.fadd.v4f32(float -0.000000e+00, <4 x float> [[PREDPHI3]])
 ; CHECK-NEXT:    br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]

llvm/test/Transforms/LoopVectorize/reduction-inloop.ll

 ; CHECK-LABEL: @reduction_sum(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP9:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_IND2:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT3:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_IND1:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT2:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
 ; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
 ; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
-; CHECK-NEXT:    [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
-; CHECK-NEXT:    [[TMP4:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[VEC_IND2]])
+; CHECK-NEXT:    [[WIDE_LOAD3:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
+; CHECK-NEXT:    [[TMP4:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[VEC_IND1]])
 ; CHECK-NEXT:    [[TMP5:%.*]] = add i32 [[TMP4]], [[VEC_PHI]]
 ; CHECK-NEXT:    [[TMP6:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD]])
 ; CHECK-NEXT:    [[TMP7:%.*]] = add i32 [[TMP6]], [[TMP5]]
-; CHECK-NEXT:    [[TMP8:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD1]])
+; CHECK-NEXT:    [[TMP8:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD3]])
 ; CHECK-NEXT:    [[TMP9]] = add i32 [[TMP8]], [[TMP7]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-NEXT:    [[VEC_IND_NEXT3]] = add <4 x i32> [[VEC_IND2]], <i32 4, i32 4, i32 4, i32 4>
+; CHECK-NEXT:    [[VEC_IND_NEXT2]] = add <4 x i32> [[VEC_IND1]], <i32 4, i32 4, i32 4, i32 4>
 ; CHECK-NEXT:    [[TMP10:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
 ; CHECK-NEXT:    br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    br label [[DOTLR_PH:%.*]]
 ; CHECK:       .lr.ph:
 ; CHECK-NEXT:    br i1 undef, label [[DOT_CRIT_EDGE]], label [[DOTLR_PH]], !llvm.loop [[LOOP5:![0-9]+]]

 ; CHECK-LABEL: @reduction_prod(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi i32 [ 1, [[VECTOR_PH]] ], [ [[TMP9:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_IND2:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT3:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_IND1:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT2:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
 ; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
 ; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
-; CHECK-NEXT:    [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
-; CHECK-NEXT:    [[TMP4:%.*]] = call i32 @llvm.vector.reduce.mul.v4i32(<4 x i32> [[VEC_IND2]])
+; CHECK-NEXT:    [[WIDE_LOAD3:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
+; CHECK-NEXT:    [[TMP4:%.*]] = call i32 @llvm.vector.reduce.mul.v4i32(<4 x i32> [[VEC_IND1]])
 ; CHECK-NEXT:    [[TMP5:%.*]] = mul i32 [[TMP4]], [[VEC_PHI]]
 ; CHECK-NEXT:    [[TMP6:%.*]] = call i32 @llvm.vector.reduce.mul.v4i32(<4 x i32> [[WIDE_LOAD]])
 ; CHECK-NEXT:    [[TMP7:%.*]] = mul i32 [[TMP6]], [[TMP5]]
-; CHECK-NEXT:    [[TMP8:%.*]] = call i32 @llvm.vector.reduce.mul.v4i32(<4 x i32> [[WIDE_LOAD1]])
+; CHECK-NEXT:    [[TMP8:%.*]] = call i32 @llvm.vector.reduce.mul.v4i32(<4 x i32> [[WIDE_LOAD3]])
 ; CHECK-NEXT:    [[TMP9]] = mul i32 [[TMP8]], [[TMP7]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-NEXT:    [[VEC_IND_NEXT3]] = add <4 x i32> [[VEC_IND2]], <i32 4, i32 4, i32 4, i32 4>
+; CHECK-NEXT:    [[VEC_IND_NEXT2]] = add <4 x i32> [[VEC_IND1]], <i32 4, i32 4, i32 4, i32 4>
 ; CHECK-NEXT:    [[TMP10:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
 ; CHECK-NEXT:    br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    br label [[DOTLR_PH:%.*]]
 ; CHECK:       .lr.ph:
 ; CHECK-NEXT:    br i1 undef, label [[DOT_CRIT_EDGE]], label [[DOTLR_PH]], !llvm.loop [[LOOP9:![0-9]+]]

 ; CHECK-LABEL: @reduction_mix(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP8:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_IND2:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT3:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_IND1:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT2:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
 ; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
 ; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
-; CHECK-NEXT:    [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
-; CHECK-NEXT:    [[TMP4:%.*]] = mul nsw <4 x i32> [[WIDE_LOAD1]], [[WIDE_LOAD]]
-; CHECK-NEXT:    [[TMP5:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[VEC_IND2]])
+; CHECK-NEXT:    [[WIDE_LOAD3:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
+; CHECK-NEXT:    [[TMP4:%.*]] = mul nsw <4 x i32> [[WIDE_LOAD3]], [[WIDE_LOAD]]
+; CHECK-NEXT:    [[TMP5:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[VEC_IND1]])
 ; CHECK-NEXT:    [[TMP6:%.*]] = add i32 [[TMP5]], [[VEC_PHI]]
 ; CHECK-NEXT:    [[TMP7:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[TMP4]])
 ; CHECK-NEXT:    [[TMP8]] = add i32 [[TMP7]], [[TMP6]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-NEXT:    [[VEC_IND_NEXT3]] = add <4 x i32> [[VEC_IND2]], <i32 4, i32 4, i32 4, i32 4>
+; CHECK-NEXT:    [[VEC_IND_NEXT2]] = add <4 x i32> [[VEC_IND1]], <i32 4, i32 4, i32 4, i32 4>
 ; CHECK-NEXT:    [[TMP9:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
 ; CHECK-NEXT:    br i1 [[TMP9]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP10:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    br label [[DOTLR_PH:%.*]]
 ; CHECK:       .lr.ph:
 ; CHECK-NEXT:    br i1 undef, label [[DOT_CRIT_EDGE]], label [[DOTLR_PH]], !llvm.loop [[LOOP11:![0-9]+]]

 define float @reduction_fadd(float* nocapture %A, float* nocapture %B) {
 ; CHECK-LABEL: @reduction_fadd(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi float [ 0.000000e+00, [[VECTOR_PH]] ], [ [[TMP7:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi float [ 0.000000e+00, [[VECTOR_PH]] ], [ [[TMP5:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds float, float* [[A:%.*]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast float* [[TMP0]] to <4 x float>*
 ; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x float>, <4 x float>* [[TMP1]], align 4
 ; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds float, float* [[B:%.*]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP3:%.*]] = bitcast float* [[TMP2]] to <4 x float>*
 ; CHECK-NEXT:    [[WIDE_LOAD1:%.*]] = load <4 x float>, <4 x float>* [[TMP3]], align 4
 ; CHECK-NEXT:    [[TMP4:%.*]] = call fast float @llvm.vector.reduce.fadd.v4f32(float [[VEC_PHI]], <4 x float> [[WIDE_LOAD]])
-; CHECK-NEXT:    [[TMP5:%.*]] = call fast float @llvm.vector.reduce.fadd.v4f32(float [[TMP4]], <4 x float> [[WIDE_LOAD1]])
+; CHECK-NEXT:    [[TMP5]] = call fast float @llvm.vector.reduce.fadd.v4f32(float [[TMP4]], <4 x float> [[WIDE_LOAD1]])
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
 ; CHECK-NEXT:    [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
 ; CHECK-NEXT:    br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP22:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:

 ; CHECK-NEXT:    [[TMP6:%.*]] = fcmp ule <4 x float> [[WIDE_LOAD1]], <float 1.000000e+00, float 1.000000e+00, float 1.000000e+00, float 1.000000e+00>
 ; CHECK-NEXT:    [[TMP7:%.*]] = fcmp ogt <4 x float> [[WIDE_LOAD]], <float 2.000000e+00, float 2.000000e+00, float 2.000000e+00, float 2.000000e+00>
 ; CHECK-NEXT:    [[TMP8:%.*]] = and <4 x i1> [[TMP5]], [[TMP6]]
 ; CHECK-NEXT:    [[TMP9:%.*]] = and <4 x i1> [[TMP8]], [[TMP7]]
 ; CHECK-NEXT:    [[TMP10:%.*]] = xor <4 x i1> [[TMP7]], <i1 true, i1 true, i1 true, i1 true>
 ; CHECK-NEXT:    [[TMP11:%.*]] = and <4 x i1> [[TMP8]], [[TMP10]]
 ; CHECK-NEXT:    [[TMP12:%.*]] = xor <4 x i1> [[TMP5]], <i1 true, i1 true, i1 true, i1 true>
 ; CHECK-NEXT:    [[PREDPHI_V:%.*]] = select <4 x i1> [[TMP9]], <4 x float> [[WIDE_LOAD1]], <4 x float> [[WIDE_LOAD]]
-; CHECK-NEXT:    [[PREDPHI:%.*]] = fadd fast <4 x float> [[VEC_PHI]], [[PREDPHI_V]]
 ; CHECK-NEXT:    [[TMP13:%.*]] = select <4 x i1> [[TMP12]], <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i1> [[TMP11]]
-; CHECK-NEXT:    [[PREDPHI3]] = select <4 x i1> [[TMP13]], <4 x float> [[VEC_PHI]], <4 x float> [[PREDPHI]]
+; CHECK-NEXT:    [[PREDPHI2:%.*]] = select <4 x i1> [[TMP13]], <4 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, <4 x float> [[PREDPHI_V]]
+; CHECK-NEXT:    [[PREDPHI3]] = fadd fast <4 x float> [[VEC_PHI]], [[PREDPHI2]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
 ; CHECK-NEXT:    [[TMP14:%.*]] = icmp eq i64 [[INDEX_NEXT]], 128
 ; CHECK-NEXT:    br i1 [[TMP14]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP32:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    [[TMP15:%.*]] = call fast float @llvm.vector.reduce.fadd.v4f32(float -0.000000e+00, <4 x float> [[PREDPHI3]])
 ; CHECK-NEXT:    br i1 true, label [[FOR_END:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]

 ; CHECK-LABEL: @reduction_predicated(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi i32 [ 0, [[VECTOR_PH]] ], [ [[TMP9:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_IND2:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT3:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_IND1:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, [[VECTOR_PH]] ], [ [[VEC_IND_NEXT2:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i32* [[TMP0]] to <4 x i32>*
 ; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP1]], align 4
 ; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i32* [[TMP2]] to <4 x i32>*
-; CHECK-NEXT:    [[WIDE_LOAD1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
-; CHECK-NEXT:    [[TMP4:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[VEC_IND2]])
+; CHECK-NEXT:    [[WIDE_LOAD3:%.*]] = load <4 x i32>, <4 x i32>* [[TMP3]], align 4
+; CHECK-NEXT:    [[TMP4:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[VEC_IND1]])
 ; CHECK-NEXT:    [[TMP5:%.*]] = add i32 [[TMP4]], [[VEC_PHI]]
 ; CHECK-NEXT:    [[TMP6:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD]])
 ; CHECK-NEXT:    [[TMP7:%.*]] = add i32 [[TMP6]], [[TMP5]]
-; CHECK-NEXT:    [[TMP8:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD1]])
+; CHECK-NEXT:    [[TMP8:%.*]] = call i32 @llvm.vector.reduce.add.v4i32(<4 x i32> [[WIDE_LOAD3]])
 ; CHECK-NEXT:    [[TMP9]] = add i32 [[TMP8]], [[TMP7]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-NEXT:    [[VEC_IND_NEXT3]] = add <4 x i32> [[VEC_IND2]], <i32 4, i32 4, i32 4, i32 4>
+; CHECK-NEXT:    [[VEC_IND_NEXT2]] = add <4 x i32> [[VEC_IND1]], <i32 4, i32 4, i32 4, i32 4>
 ; CHECK-NEXT:    [[TMP10:%.*]] = icmp eq i64 [[INDEX_NEXT]], 256
 ; CHECK-NEXT:    br i1 [[TMP10]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP34:![0-9]+]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    br i1 true, label [[DOT_CRIT_EDGE:%.*]], label [[SCALAR_PH]]
 ; CHECK:       scalar.ph:
 ; CHECK-NEXT:    br label [[DOTLR_PH:%.*]]
 ; CHECK:       .lr.ph:
 ; CHECK-NEXT:    br i1 undef, label [[DOT_CRIT_EDGE]], label [[DOTLR_PH]], !llvm.loop [[LOOP35:![0-9]+]]

llvm/test/Transforms/PhaseOrdering/AArch64/hoisting-sinking-required-for-vectorization.ll

 ; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq <4 x i32> [[WIDE_LOAD]], <i32 20, i32 20, i32 20, i32 20>
 ; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds float, float* [[A]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP6:%.*]] = bitcast float* [[TMP5]] to <4 x float>*
 ; CHECK-NEXT:    [[WIDE_LOAD14:%.*]] = load <4 x float>, <4 x float>* [[TMP6]], align 4, !alias.scope !11
 ; CHECK-NEXT:    [[TMP7:%.*]] = fmul <4 x float> [[WIDE_LOAD14]], [[BROADCAST_SPLAT]]
 ; CHECK-NEXT:    [[TMP8:%.*]] = getelementptr inbounds float, float* [[B]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP9:%.*]] = bitcast float* [[TMP8]] to <4 x float>*
 ; CHECK-NEXT:    [[WIDE_LOAD15:%.*]] = load <4 x float>, <4 x float>* [[TMP9]], align 4, !alias.scope !13, !noalias !15
-; CHECK-NEXT:    [[TMP10:%.*]] = fadd <4 x float> [[TMP7]], [[WIDE_LOAD15]]
-; CHECK-NEXT:    [[PREDPHI:%.*]] = select <4 x i1> [[TMP4]], <4 x float> [[TMP7]], <4 x float> [[TMP10]]
+; CHECK-NEXT:    [[TMP10:%.*]] = select <4 x i1> [[TMP4]], <4 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, <4 x float> [[WIDE_LOAD15]]
+; CHECK-NEXT:    [[PREDPHI:%.*]] = fadd <4 x float> [[TMP7]], [[TMP10]]
 ; CHECK-NEXT:    [[TMP11:%.*]] = bitcast float* [[TMP8]] to <4 x float>*
 ; CHECK-NEXT:    store <4 x float> [[PREDPHI]], <4 x float>* [[TMP11]], align 4, !alias.scope !13, !noalias !15
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
 ; CHECK-NEXT:    [[TMP12:%.*]] = icmp eq i64 [[INDEX_NEXT]], 10000
 ; CHECK-NEXT:    br i1 [[TMP12]], label [[EXIT:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP16:![0-9]+]]
 ; CHECK:       loop.body:
 ; CHECK-NEXT:    [[IV1:%.*]] = phi i64 [ [[IV_NEXT:%.*]], [[LOOP_LATCH:%.*]] ], [ 0, [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    [[C_GEP:%.*]] = getelementptr inbounds i32, i32* [[C]], i64 [[IV1]]