[InstCombine] Don't sink the fdiv from (fmul (fdiv 1.0, %x), %y) if the fdiv isn't in the same basic block as the fmul

[craig.topper]

Summary

Loop invariant code motion can pull an fdiv out of by putting a reciprocal outside and using an fmul inside. InstCombine shouldn't reverse that.

I just did a simple basic block parent check rather than checking for the loop. I can call getLoop and isInvariant if we think that would be better.

[craig.topper]

Add a comment describing the exception

[lebedev.ri]

When does that happen?
-O3 seems to be fine: https://godbolt.org/z/1K3r56
Do we have InstCombine invocations after last LICM?

[craig.topper]

In D87479#2266582, @lebedev.ri wrote:

When does that happen?
-O3 seems to be fine: https://godbolt.org/z/1K3r56
Do we have InstCombine invocations after last LICM?

We found it in our internal code base so maybe there’s something different about our pipeline. But still seems like we shouldn’t have two passes making opposite choices.

[lebedev.ri]

In D87479#2266721, @craig.topper wrote:

In D87479#2266582, @lebedev.ri wrote:

When does that happen?
-O3 seems to be fine: https://godbolt.org/z/1K3r56
Do we have InstCombine invocations after last LICM?

We found it in our internal code base so maybe there’s something different about our pipeline.

But still seems like we shouldn’t have two passes making opposite choices.

I think this isn't an exception, there's a lot of opposite transforms.

[craig.topper]

In D87479#2266728, @lebedev.ri wrote:

In D87479#2266721, @craig.topper wrote:

In D87479#2266582, @lebedev.ri wrote:

When does that happen?
-O3 seems to be fine: https://godbolt.org/z/1K3r56
Do we have InstCombine invocations after last LICM?

We found it in our internal code base so maybe there’s something different about our pipeline.

But still seems like we shouldn’t have two passes making opposite choices.

I think this isn't an exception, there's a lot of opposite transforms.

I don't know if it helps, but the last LICM that fixes it is disabled with -fno-unroll-loops https://godbolt.org/z/dxjMxb Though for this simple case MachineLICM recovers it. But MachineLICM is much weaker than IR LICM on nested loops.

[spatel]

This sounds very much like what happened here:
https://llvm.org/PR46115
https://github.com/JuliaLang/julia/issues/36084

So either we need a much bigger change (instcombine should not do code sinking), or we try to enforce not running instcombine after LICM somehow?

[craig.topper]

We don't seem to run LICM after InstCombine in the LTO pipeline if I'm reading the pass manager correctly.

[craig.topper]

Fix copy paste mistake. Update phase ordering test that is now affected.

[spatel]

In D87479#2266829, @craig.topper wrote:

We don't seem to run LICM after InstCombine in the LTO pipeline if I'm reading the pass manager correctly.

That seems like the root bug...although a 1st hack at changing that didn't result in any regression test fails in llvm/test/Other/* .
If we are inverting this fdiv pattern, then aren't we doing that to all kinds of other loop invariant opportunities too?

[spatel]

In D87479#2267835, @spatel wrote:

In D87479#2266829, @craig.topper wrote:

We don't seem to run LICM after InstCombine in the LTO pipeline if I'm reading the pass manager correctly.

That seems like the root bug...although a 1st hack at changing that didn't result in any regression test fails in llvm/test/Other/* .
If we are inverting this fdiv pattern, then aren't we doing that to all kinds of other loop invariant opportunities too?

Actually, we do have some coverage for the new pass manager at least. I got that to wiggle with:

diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index bae84784628..acc937cb202 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -1576,6 +1576,9 @@ PassBuilder::buildLTODefaultPipeline(OptimizationLevel Level, bool DebugLogging,
   MainFPM.addPass(InstCombinePass());
   invokePeepholeEPCallbacks(MainFPM, Level);
   MainFPM.addPass(JumpThreadingPass(/*InsertFreezeWhenUnfoldingSelect*/ true));
+  MainFPM.addPass(createFunctionToLoopPassAdaptor(
+      LICMPass(PTO.LicmMssaOptCap, PTO.LicmMssaNoAccForPromotionCap),
+      EnableMSSALoopDependency, DebugLogging));
   MPM.addPass(createModuleToFunctionPassAdaptor(std::move(MainFPM)));
 
   // Create a function that performs CFI checks for cross-DSO calls with

[craig.topper]

In D87479#2267835, @spatel wrote:

In D87479#2266829, @craig.topper wrote:

We don't seem to run LICM after InstCombine in the LTO pipeline if I'm reading the pass manager correctly.

That seems like the root bug...although a 1st hack at changing that didn't result in any regression test fails in llvm/test/Other/* .
If we are inverting this fdiv pattern, then aren't we doing that to all kinds of other loop invariant opportunities too?

We likely are messing up other opportunities, but maybe their cost isn't as high as division so no one has noticed.

The case we spotted this on wasn't with LTO. So maybe we have another problem in our internal pipeline like Julia had.

It also seems to affect vectorization as the PhaseOrder test is showing. And we don't get it right with -fno-unroll-loops. Are those worth fixing?

[lebedev.ri]

I personally really think this should be viewed as phase ordering issue, that should be addressed by adding/moving passes in pipeline.

[spatel]

In D87479#2268474, @lebedev.ri wrote:

I personally really think this should be viewed as phase ordering issue, that should be addressed by adding/moving pases in pipeline.

I agree. We have 3 likely problems already identified in this discussion: instcombine vs. LICM in LTO, instcombine vs. (no-)unroll, instcombine vs. loop-vectorizer.
The common opponent is instcombine of course because it does sinking transforms without cost models.

[xbolva00]

Maybe PR4268 is similar issue..

[spatel]

In D87479#2268567, @xbolva00 wrote:

Maybe PR4268 is similar issue..

Wrong bug number?

[xbolva00]

Ouch, sorry

https://bugs.llvm.org/show_bug.cgi?id=42681

[lebedev.ri]

Just marking as reviewed since the review appears to be stalling.

[RKSimon]

@craig.topper Are you still looking at this?

[craig.topper]

In D87479#2631190, @RKSimon wrote:

@craig.topper Are you still looking at this?

No, not since I switched employers.

[RKSimon]

In D87479#2632032, @craig.topper wrote:

In D87479#2631190, @RKSimon wrote:

@craig.topper Are you still looking at this?

No, not since I switched employers.

Would it be OK if myself or @spatel commandeered this?

[craig.topper]

In D87479#2632332, @RKSimon wrote:

In D87479#2632032, @craig.topper wrote:

In D87479#2631190, @RKSimon wrote:

@craig.topper Are you still looking at this?

No, not since I switched employers.

Would it be OK if myself or @spatel commandeered this?

Absolutely.

[spatel]

The -Ofast -fno-unroll-loops example still shows up in the 12.0 release, but is fixed in trunk:
https://godbolt.org/z/4vss8qrTG

I didn't track down the commit that changed it, but I have added a regression test, so it will stick:
31e997fda1c6

[spatel]

In D87479#2694818, @spatel wrote:

https://godbolt.org/z/4vss8qrTG

The x86 asm also shows an interesting trade-off: SDAG could form a reciprocal estimate when the fdiv was sunk back into the loop...but it couldn't hoist the refinement math, so that's likely worse perf.

[lebedev.ri]

This review seems to be stuck/dead, consider abandoning if no longer relevant.

[craig.topper]

llvm/lib/Transforms/InstCombine/InstCombineMulDivRem.cpp

531

I found another variation of this

(X/Y) outside loop
Mul with Z inside loop.

->

X*Z ends up inside loop
/ Y ends up in loop

->

recip of Y outside loop

X*Z in loop
mul with reciprocal of Y inside loop.

So we went from an fdiv outside loop and one fmul inside the loop to 1 div outside the loop and 2 fmuls inside the loop.

[spatel]

llvm/lib/Transforms/InstCombine/InstCombineMulDivRem.cpp

531

Does this model it:
https://godbolt.org/z/W4Ta9jzEn ?
...although MachineLICM is managing to pull the fmuls out of the loop there, so there must be some other complication?

[craig.topper]

llvm/lib/Transforms/InstCombine/InstCombineMulDivRem.cpp

531

MachineLICM doesn't fix it on pure scalar.

https://godbolt.org/z/zGvxxqaTP

llvm/lib/Transforms/InstCombine/InstCombineMulDivRem.cpp

 //===- InstCombineMulDivRem.cpp -------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
if (match(Op1, m_Constant(C)) && C->isFiniteNonZeroFP()) {
       if (match(Op0, m_OneUse(m_FSub(m_Constant(C1), m_Value(X))))) {
         // (C1 - X) * C --> (C * C1) - (X * C)
         Constant *CC1 = ConstantExpr::getFMul(C, C1);
         Value *XC = Builder.CreateFMulFMF(X, C, &I);
         return BinaryOperator::CreateFSubFMF(CC1, XC, &I);
       }
     }
 
-    Value *Z;
-    if (match(&I, m_c_FMul(m_OneUse(m_FDiv(m_Value(X), m_Value(Y))),
-                           m_Value(Z)))) {
-      // Sink division: (X / Y) * Z --> (X * Z) / Y
-      Value *NewFMul = Builder.CreateFMulFMF(X, Z, &I);
+    // Don't sink if the fdiv is a reciprocal in a different basic block as this
+    // might pull a division into a loop undoing a transform from LICM.
+    if (match(Op0, m_OneUse(m_FDiv(m_Value(X), m_Value(Y)))) &&
+        (!match(X, m_FPOne()) ||
+         cast<Instruction>(Op0)->getParent() == I.getParent())) {
+      Value *NewFMul = Builder.CreateFMulFMF(X, Op1, &I);
+      return BinaryOperator::CreateFDivFMF(NewFMul, Y, &I);
+    }
+    if (match(Op1, m_OneUse(m_FDiv(m_Value(X), m_Value(Y)))) &&
+        (!match(X, m_FPOne()) ||
+         cast<Instruction>(Op1)->getParent() == I.getParent())) {
+      Value *NewFMul = Builder.CreateFMulFMF(X, Op0, &I);
       return BinaryOperator::CreateFDivFMF(NewFMul, Y, &I);
     }
 
     // sqrt(X) * sqrt(Y) -> sqrt(X * Y)
     // nnan disallows the possibility of returning a number if both operands are
     // negative (in that case, we should return NaN).
     if (I.hasNoNaNs() &&
         match(Op0, m_OneUse(m_Intrinsic<Intrinsic::sqrt>(m_Value(X)))) &&

llvm/test/Transforms/InstCombine/fmul.ll

 ; CHECK-NEXT:    ret float [[MUL]]
 ;
   %div = fdiv float %x, 42.0
   call void @use_f32(float %div)
   %mul = fmul reassoc float %div, %y
   ret float %mul
 }
 
+; Make sure we don't sink this invariant fdiv into the loop.
+define void @fmul_loop_invariant_fdiv(float* %a, float %x) {
+; CHECK-LABEL: @fmul_loop_invariant_fdiv(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = fdiv fast float 1.000000e+00, [[X:%.*]]
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.cond.cleanup:
+; CHECK-NEXT:    ret void
+; CHECK:       for.body:
+; CHECK-NEXT:    [[I_08:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[IDXPROM:%.*]] = zext i32 [[I_08]] to i64
+; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds float, float* [[A:%.*]], i64 [[IDXPROM]]
+; CHECK-NEXT:    [[TMP1:%.*]] = load float, float* [[ARRAYIDX]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul fast float [[TMP1]], [[TMP0]]
+; CHECK-NEXT:    store float [[TMP2]], float* [[ARRAYIDX]], align 4
+; CHECK-NEXT:    [[INC]] = add nuw nsw i32 [[I_08]], 1
+; CHECK-NEXT:    [[CMP_NOT:%.*]] = icmp eq i32 [[INC]], 1024
+; CHECK-NEXT:    br i1 [[CMP_NOT]], label [[FOR_COND_CLEANUP:%.*]], label [[FOR_BODY]]
+;
+entry:
+  %0 = fdiv fast float 1.000000e+00, %x
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.body
+  ret void
+
+for.body:                                         ; preds = %entry, %for.body
+  %i.08 = phi i32 [ 0, %entry ], [ %inc, %for.body ]
+  %idxprom = zext i32 %i.08 to i64
+  %arrayidx = getelementptr inbounds float, float* %a, i64 %idxprom
+  %1 = load float, float* %arrayidx, align 4
+  %2 = fmul fast float %1, %0
+  store float %2, float* %arrayidx, align 4
+  %inc = add nuw nsw i32 %i.08, 1
+  %cmp.not = icmp eq i32 %inc, 1024
+  br i1 %cmp.not, label %for.cond.cleanup, label %for.body
+}
+
 ; Avoid infinite looping by moving negation out of a constant expression.
 
 @g = external global {[2 x i8*]}, align 1
 
 define double @fmul_negated_constant_expression(double %x) {
 ; CHECK-LABEL: @fmul_negated_constant_expression(
 ; CHECK-NEXT:    [[R:%.*]] = fmul double [[X:%.*]], fsub (double -0.000000e+00, double bitcast (i64 ptrtoint (i8** getelementptr inbounds ({ [2 x i8*] }, { [2 x i8*] }* @g, i64 0, inrange i32 0, i64 2) to i64) to double))
 ; CHECK-NEXT:    ret double [[R]]

llvm/test/Transforms/PhaseOrdering/X86/vdiv.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -O3 -S                                        | FileCheck %s
 ; RUN: opt < %s -passes='default<O3>' -aa-pipeline=default -S | FileCheck %s
 
 ; Test that IR is optimal after vectorization/unrolling/CSE/canonicalization.
 ; In particular, there should be no fdivs inside loops because that is expensive.
 
 ; TODO: There is a CSE opportunity to reduce the hoisted fdivs after vectorization/unrolling.
 ; PR46115 - https://bugs.llvm.org/PR46115
 
 target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.15.0"
 
 define void @vdiv(double* %x, double* %y, double %a, i32 %N) #0 {
 ; CHECK-LABEL: @vdiv(
 ; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[DIV:%.*]] = fdiv fast double 1.000000e+00, [[A:%.*]]
 ; CHECK-NEXT:    [[CMP1:%.*]] = icmp sgt i32 [[N:%.*]], 0
-; CHECK-NEXT:    br i1 [[CMP1]], label [[FOR_BODY_LR_PH:%.*]], label [[FOR_END:%.*]]
-; CHECK:       for.body.lr.ph:
+; CHECK-NEXT:    br i1 [[CMP1]], label [[FOR_BODY_PREHEADER:%.*]], label [[FOR_END:%.*]]
+; CHECK:       for.body.preheader:
 ; CHECK-NEXT:    [[WIDE_TRIP_COUNT:%.*]] = zext i32 [[N]] to i64
-; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i32 [[N]], 4
-; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[FOR_BODY_PREHEADER:%.*]], label [[VECTOR_MEMCHECK:%.*]]
+; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i32 [[N]], 16
+; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[FOR_BODY_PREHEADER17:%.*]], label [[VECTOR_MEMCHECK:%.*]]
 ; CHECK:       vector.memcheck:
 ; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr double, double* [[X:%.*]], i64 [[WIDE_TRIP_COUNT]]
 ; CHECK-NEXT:    [[SCEVGEP6:%.*]] = getelementptr double, double* [[Y:%.*]], i64 [[WIDE_TRIP_COUNT]]
 ; CHECK-NEXT:    [[BOUND0:%.*]] = icmp ugt double* [[SCEVGEP6]], [[X]]
 ; CHECK-NEXT:    [[BOUND1:%.*]] = icmp ugt double* [[SCEVGEP]], [[Y]]
 ; CHECK-NEXT:    [[FOUND_CONFLICT:%.*]] = and i1 [[BOUND0]], [[BOUND1]]
-; CHECK-NEXT:    br i1 [[FOUND_CONFLICT]], label [[FOR_BODY_PREHEADER]], label [[VECTOR_PH:%.*]]
+; CHECK-NEXT:    br i1 [[FOUND_CONFLICT]], label [[FOR_BODY_PREHEADER17]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
-; CHECK-NEXT:    [[N_VEC:%.*]] = and i64 [[WIDE_TRIP_COUNT]], 4294967292
-; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x double> undef, double [[A:%.*]], i32 0
+; CHECK-NEXT:    [[N_VEC:%.*]] = and i64 [[WIDE_TRIP_COUNT]], 4294967280
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x double> undef, double [[DIV]], i32 0
 ; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x double> [[BROADCAST_SPLATINSERT]], <4 x double> undef, <4 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP0:%.*]] = add nsw i64 [[N_VEC]], -4
-; CHECK-NEXT:    [[TMP1:%.*]] = lshr exact i64 [[TMP0]], 2
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT11:%.*]] = insertelement <4 x double> undef, double [[DIV]], i32 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT12:%.*]] = shufflevector <4 x double> [[BROADCAST_SPLATINSERT11]], <4 x double> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT13:%.*]] = insertelement <4 x double> undef, double [[DIV]], i32 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT14:%.*]] = shufflevector <4 x double> [[BROADCAST_SPLATINSERT13]], <4 x double> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[BROADCAST_SPLATINSERT15:%.*]] = insertelement <4 x double> undef, double [[DIV]], i32 0
+; CHECK-NEXT:    [[BROADCAST_SPLAT16:%.*]] = shufflevector <4 x double> [[BROADCAST_SPLATINSERT15]], <4 x double> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP0:%.*]] = add nsw i64 [[N_VEC]], -16
+; CHECK-NEXT:    [[TMP1:%.*]] = lshr exact i64 [[TMP0]], 4
 ; CHECK-NEXT:    [[TMP2:%.*]] = add nuw nsw i64 [[TMP1]], 1
-; CHECK-NEXT:    [[XTRAITER:%.*]] = and i64 [[TMP2]], 3
-; CHECK-NEXT:    [[TMP3:%.*]] = icmp ult i64 [[TMP0]], 12
+; CHECK-NEXT:    [[XTRAITER18:%.*]] = and i64 [[TMP2]], 1
+; CHECK-NEXT:    [[TMP3:%.*]] = icmp eq i64 [[TMP0]], 0
 ; CHECK-NEXT:    br i1 [[TMP3]], label [[MIDDLE_BLOCK_UNR_LCSSA:%.*]], label [[VECTOR_PH_NEW:%.*]]
 ; CHECK:       vector.ph.new:
-; CHECK-NEXT:    [[UNROLL_ITER:%.*]] = and i64 [[TMP2]], 9223372036854775804
-; CHECK-NEXT:    [[TMP4:%.*]] = fdiv fast <4 x double> <double 1.000000e+00, double 1.000000e+00, double 1.000000e+00, double 1.000000e+00>, [[BROADCAST_SPLAT]]
-; CHECK-NEXT:    [[TMP5:%.*]] = fdiv fast <4 x double> <double 1.000000e+00, double 1.000000e+00, double 1.000000e+00, double 1.000000e+00>, [[BROADCAST_SPLAT]]
-; CHECK-NEXT:    [[TMP6:%.*]] = fdiv fast <4 x double> <double 1.000000e+00, double 1.000000e+00, double 1.000000e+00, double 1.000000e+00>, [[BROADCAST_SPLAT]]
-; CHECK-NEXT:    [[TMP7:%.*]] = fdiv fast <4 x double> <double 1.000000e+00, double 1.000000e+00, double 1.000000e+00, double 1.000000e+00>, [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[UNROLL_ITER:%.*]] = and i64 [[TMP2]], 2305843009213693950
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
-; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH_NEW]] ], [ [[INDEX_NEXT_3:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[NITER:%.*]] = phi i64 [ [[UNROLL_ITER]], [[VECTOR_PH_NEW]] ], [ [[NITER_NSUB_3:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[TMP8:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDEX]]
+; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH_NEW]] ], [ [[INDEX_NEXT_1:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[NITER:%.*]] = phi i64 [ [[UNROLL_ITER]], [[VECTOR_PH_NEW]] ], [ [[NITER_NSUB_1:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[TMP4:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDEX]]
+; CHECK-NEXT:    [[TMP5:%.*]] = bitcast double* [[TMP4]] to <4 x double>*
+; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x double>, <4 x double>* [[TMP5]], align 8, [[TBAA3:!tbaa !.*]], !alias.scope !7
+; CHECK-NEXT:    [[TMP6:%.*]] = getelementptr inbounds double, double* [[TMP4]], i64 4
+; CHECK-NEXT:    [[TMP7:%.*]] = bitcast double* [[TMP6]] to <4 x double>*
+; CHECK-NEXT:    [[WIDE_LOAD8:%.*]] = load <4 x double>, <4 x double>* [[TMP7]], align 8, [[TBAA3]], !alias.scope !7
+; CHECK-NEXT:    [[TMP8:%.*]] = getelementptr inbounds double, double* [[TMP4]], i64 8
 ; CHECK-NEXT:    [[TMP9:%.*]] = bitcast double* [[TMP8]] to <4 x double>*
-; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <4 x double>, <4 x double>* [[TMP9]], align 8, !tbaa !3, !alias.scope !7
-; CHECK-NEXT:    [[TMP10:%.*]] = fmul fast <4 x double> [[WIDE_LOAD]], [[TMP4]]
-; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDEX]]
-; CHECK-NEXT:    [[TMP12:%.*]] = bitcast double* [[TMP11]] to <4 x double>*
-; CHECK-NEXT:    store <4 x double> [[TMP10]], <4 x double>* [[TMP12]], align 8, !tbaa !3, !alias.scope !10, !noalias !7
-; CHECK-NEXT:    [[INDEX_NEXT:%.*]] = or i64 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP13:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDEX_NEXT]]
-; CHECK-NEXT:    [[TMP14:%.*]] = bitcast double* [[TMP13]] to <4 x double>*
-; CHECK-NEXT:    [[WIDE_LOAD_1:%.*]] = load <4 x double>, <4 x double>* [[TMP14]], align 8, !tbaa !3, !alias.scope !7
-; CHECK-NEXT:    [[TMP15:%.*]] = fmul fast <4 x double> [[WIDE_LOAD_1]], [[TMP5]]
-; CHECK-NEXT:    [[TMP16:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDEX_NEXT]]
+; CHECK-NEXT:    [[WIDE_LOAD9:%.*]] = load <4 x double>, <4 x double>* [[TMP9]], align 8, [[TBAA3]], !alias.scope !7
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr inbounds double, double* [[TMP4]], i64 12
+; CHECK-NEXT:    [[TMP11:%.*]] = bitcast double* [[TMP10]] to <4 x double>*
+; CHECK-NEXT:    [[WIDE_LOAD10:%.*]] = load <4 x double>, <4 x double>* [[TMP11]], align 8, [[TBAA3]], !alias.scope !7
+; CHECK-NEXT:    [[TMP12:%.*]] = fmul fast <4 x double> [[WIDE_LOAD]], [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[TMP13:%.*]] = fmul fast <4 x double> [[WIDE_LOAD8]], [[BROADCAST_SPLAT12]]
+; CHECK-NEXT:    [[TMP14:%.*]] = fmul fast <4 x double> [[WIDE_LOAD9]], [[BROADCAST_SPLAT14]]
+; CHECK-NEXT:    [[TMP15:%.*]] = fmul fast <4 x double> [[WIDE_LOAD10]], [[BROADCAST_SPLAT16]]
+; CHECK-NEXT:    [[TMP16:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDEX]]
 ; CHECK-NEXT:    [[TMP17:%.*]] = bitcast double* [[TMP16]] to <4 x double>*
-; CHECK-NEXT:    store <4 x double> [[TMP15]], <4 x double>* [[TMP17]], align 8, !tbaa !3, !alias.scope !10, !noalias !7
-; CHECK-NEXT:    [[INDEX_NEXT_1:%.*]] = or i64 [[INDEX]], 8
-; CHECK-NEXT:    [[TMP18:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDEX_NEXT_1]]
+; CHECK-NEXT:    store <4 x double> [[TMP12]], <4 x double>* [[TMP17]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    [[TMP18:%.*]] = getelementptr inbounds double, double* [[TMP16]], i64 4
 ; CHECK-NEXT:    [[TMP19:%.*]] = bitcast double* [[TMP18]] to <4 x double>*
-; CHECK-NEXT:    [[WIDE_LOAD_2:%.*]] = load <4 x double>, <4 x double>* [[TMP19]], align 8, !tbaa !3, !alias.scope !7
-; CHECK-NEXT:    [[TMP20:%.*]] = fmul fast <4 x double> [[WIDE_LOAD_2]], [[TMP6]]
-; CHECK-NEXT:    [[TMP21:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDEX_NEXT_1]]
-; CHECK-NEXT:    [[TMP22:%.*]] = bitcast double* [[TMP21]] to <4 x double>*
-; CHECK-NEXT:    store <4 x double> [[TMP20]], <4 x double>* [[TMP22]], align 8, !tbaa !3, !alias.scope !10, !noalias !7
-; CHECK-NEXT:    [[INDEX_NEXT_2:%.*]] = or i64 [[INDEX]], 12
-; CHECK-NEXT:    [[TMP23:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDEX_NEXT_2]]
-; CHECK-NEXT:    [[TMP24:%.*]] = bitcast double* [[TMP23]] to <4 x double>*
-; CHECK-NEXT:    [[WIDE_LOAD_3:%.*]] = load <4 x double>, <4 x double>* [[TMP24]], align 8, !tbaa !3, !alias.scope !7
-; CHECK-NEXT:    [[TMP25:%.*]] = fmul fast <4 x double> [[WIDE_LOAD_3]], [[TMP7]]
-; CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDEX_NEXT_2]]
+; CHECK-NEXT:    store <4 x double> [[TMP13]], <4 x double>* [[TMP19]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    [[TMP20:%.*]] = getelementptr inbounds double, double* [[TMP16]], i64 8
+; CHECK-NEXT:    [[TMP21:%.*]] = bitcast double* [[TMP20]] to <4 x double>*
+; CHECK-NEXT:    store <4 x double> [[TMP14]], <4 x double>* [[TMP21]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    [[TMP22:%.*]] = getelementptr inbounds double, double* [[TMP16]], i64 12
+; CHECK-NEXT:    [[TMP23:%.*]] = bitcast double* [[TMP22]] to <4 x double>*
+; CHECK-NEXT:    store <4 x double> [[TMP15]], <4 x double>* [[TMP23]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    [[INDEX_NEXT:%.*]] = or i64 [[INDEX]], 16
+; CHECK-NEXT:    [[TMP24:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDEX_NEXT]]
+; CHECK-NEXT:    [[TMP25:%.*]] = bitcast double* [[TMP24]] to <4 x double>*
+; CHECK-NEXT:    [[WIDE_LOAD_1:%.*]] = load <4 x double>, <4 x double>* [[TMP25]], align 8, [[TBAA3]], !alias.scope !7
+; CHECK-NEXT:    [[TMP26:%.*]] = getelementptr inbounds double, double* [[TMP24]], i64 4
 ; CHECK-NEXT:    [[TMP27:%.*]] = bitcast double* [[TMP26]] to <4 x double>*
-; CHECK-NEXT:    store <4 x double> [[TMP25]], <4 x double>* [[TMP27]], align 8, !tbaa !3, !alias.scope !10, !noalias !7
-; CHECK-NEXT:    [[INDEX_NEXT_3]] = add i64 [[INDEX]], 16
-; CHECK-NEXT:    [[NITER_NSUB_3]] = add i64 [[NITER]], -4
-; CHECK-NEXT:    [[NITER_NCMP_3:%.*]] = icmp eq i64 [[NITER_NSUB_3]], 0
-; CHECK-NEXT:    br i1 [[NITER_NCMP_3]], label [[MIDDLE_BLOCK_UNR_LCSSA]], label [[VECTOR_BODY]], !llvm.loop !12
+; CHECK-NEXT:    [[WIDE_LOAD8_1:%.*]] = load <4 x double>, <4 x double>* [[TMP27]], align 8, [[TBAA3]], !alias.scope !7
+; CHECK-NEXT:    [[TMP28:%.*]] = getelementptr inbounds double, double* [[TMP24]], i64 8
+; CHECK-NEXT:    [[TMP29:%.*]] = bitcast double* [[TMP28]] to <4 x double>*
+; CHECK-NEXT:    [[WIDE_LOAD9_1:%.*]] = load <4 x double>, <4 x double>* [[TMP29]], align 8, [[TBAA3]], !alias.scope !7
+; CHECK-NEXT:    [[TMP30:%.*]] = getelementptr inbounds double, double* [[TMP24]], i64 12
+; CHECK-NEXT:    [[TMP31:%.*]] = bitcast double* [[TMP30]] to <4 x double>*
+; CHECK-NEXT:    [[WIDE_LOAD10_1:%.*]] = load <4 x double>, <4 x double>* [[TMP31]], align 8, [[TBAA3]], !alias.scope !7
+; CHECK-NEXT:    [[TMP32:%.*]] = fmul fast <4 x double> [[WIDE_LOAD_1]], [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[TMP33:%.*]] = fmul fast <4 x double> [[WIDE_LOAD8_1]], [[BROADCAST_SPLAT12]]
+; CHECK-NEXT:    [[TMP34:%.*]] = fmul fast <4 x double> [[WIDE_LOAD9_1]], [[BROADCAST_SPLAT14]]
+; CHECK-NEXT:    [[TMP35:%.*]] = fmul fast <4 x double> [[WIDE_LOAD10_1]], [[BROADCAST_SPLAT16]]
+; CHECK-NEXT:    [[TMP36:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDEX_NEXT]]
+; CHECK-NEXT:    [[TMP37:%.*]] = bitcast double* [[TMP36]] to <4 x double>*
+; CHECK-NEXT:    store <4 x double> [[TMP32]], <4 x double>* [[TMP37]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    [[TMP38:%.*]] = getelementptr inbounds double, double* [[TMP36]], i64 4
+; CHECK-NEXT:    [[TMP39:%.*]] = bitcast double* [[TMP38]] to <4 x double>*
+; CHECK-NEXT:    store <4 x double> [[TMP33]], <4 x double>* [[TMP39]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    [[TMP40:%.*]] = getelementptr inbounds double, double* [[TMP36]], i64 8
+; CHECK-NEXT:    [[TMP41:%.*]] = bitcast double* [[TMP40]] to <4 x double>*
+; CHECK-NEXT:    store <4 x double> [[TMP34]], <4 x double>* [[TMP41]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    [[TMP42:%.*]] = getelementptr inbounds double, double* [[TMP36]], i64 12
+; CHECK-NEXT:    [[TMP43:%.*]] = bitcast double* [[TMP42]] to <4 x double>*
+; CHECK-NEXT:    store <4 x double> [[TMP35]], <4 x double>* [[TMP43]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    [[INDEX_NEXT_1]] = add i64 [[INDEX]], 32
+; CHECK-NEXT:    [[NITER_NSUB_1]] = add i64 [[NITER]], -2
+; CHECK-NEXT:    [[NITER_NCMP_1:%.*]] = icmp eq i64 [[NITER_NSUB_1]], 0
+; CHECK-NEXT:    br i1 [[NITER_NCMP_1]], label [[MIDDLE_BLOCK_UNR_LCSSA]], label [[VECTOR_BODY]], [[LOOP12:!llvm.loop !.*]]
 ; CHECK:       middle.block.unr-lcssa:
-; CHECK-NEXT:    [[INDEX_UNR:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT_3]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[LCMP_MOD:%.*]] = icmp eq i64 [[XTRAITER]], 0
-; CHECK-NEXT:    br i1 [[LCMP_MOD]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY_EPIL_PREHEADER:%.*]]
-; CHECK:       vector.body.epil.preheader:
-; CHECK-NEXT:    [[TMP28:%.*]] = fdiv fast <4 x double> <double 1.000000e+00, double 1.000000e+00, double 1.000000e+00, double 1.000000e+00>, [[BROADCAST_SPLAT]]
-; CHECK-NEXT:    br label [[VECTOR_BODY_EPIL:%.*]]
+; CHECK-NEXT:    [[INDEX_UNR:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT_1]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[LCMP_MOD19_NOT:%.*]] = icmp eq i64 [[XTRAITER18]], 0
+; CHECK-NEXT:    br i1 [[LCMP_MOD19_NOT]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY_EPIL:%.*]]
 ; CHECK:       vector.body.epil:
-; CHECK-NEXT:    [[INDEX_EPIL:%.*]] = phi i64 [ [[INDEX_UNR]], [[VECTOR_BODY_EPIL_PREHEADER]] ], [ [[INDEX_NEXT_EPIL:%.*]], [[VECTOR_BODY_EPIL]] ]
-; CHECK-NEXT:    [[EPIL_ITER:%.*]] = phi i64 [ [[XTRAITER]], [[VECTOR_BODY_EPIL_PREHEADER]] ], [ [[EPIL_ITER_SUB:%.*]], [[VECTOR_BODY_EPIL]] ]
-; CHECK-NEXT:    [[TMP29:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDEX_EPIL]]
-; CHECK-NEXT:    [[TMP30:%.*]] = bitcast double* [[TMP29]] to <4 x double>*
-; CHECK-NEXT:    [[WIDE_LOAD_EPIL:%.*]] = load <4 x double>, <4 x double>* [[TMP30]], align 8, !tbaa !3, !alias.scope !7
-; CHECK-NEXT:    [[TMP31:%.*]] = fmul fast <4 x double> [[WIDE_LOAD_EPIL]], [[TMP28]]
-; CHECK-NEXT:    [[TMP32:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDEX_EPIL]]
-; CHECK-NEXT:    [[TMP33:%.*]] = bitcast double* [[TMP32]] to <4 x double>*
-; CHECK-NEXT:    store <4 x double> [[TMP31]], <4 x double>* [[TMP33]], align 8, !tbaa !3, !alias.scope !10, !noalias !7
-; CHECK-NEXT:    [[INDEX_NEXT_EPIL]] = add i64 [[INDEX_EPIL]], 4
-; CHECK-NEXT:    [[EPIL_ITER_SUB]] = add i64 [[EPIL_ITER]], -1
-; CHECK-NEXT:    [[EPIL_ITER_CMP:%.*]] = icmp eq i64 [[EPIL_ITER_SUB]], 0
-; CHECK-NEXT:    br i1 [[EPIL_ITER_CMP]], label [[MIDDLE_BLOCK]], label [[VECTOR_BODY_EPIL]], !llvm.loop !14
+; CHECK-NEXT:    [[TMP44:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDEX_UNR]]
+; CHECK-NEXT:    [[TMP45:%.*]] = bitcast double* [[TMP44]] to <4 x double>*
+; CHECK-NEXT:    [[WIDE_LOAD_EPIL:%.*]] = load <4 x double>, <4 x double>* [[TMP45]], align 8, [[TBAA3]], !alias.scope !7
+; CHECK-NEXT:    [[TMP46:%.*]] = getelementptr inbounds double, double* [[TMP44]], i64 4
+; CHECK-NEXT:    [[TMP47:%.*]] = bitcast double* [[TMP46]] to <4 x double>*
+; CHECK-NEXT:    [[WIDE_LOAD8_EPIL:%.*]] = load <4 x double>, <4 x double>* [[TMP47]], align 8, [[TBAA3]], !alias.scope !7
+; CHECK-NEXT:    [[TMP48:%.*]] = getelementptr inbounds double, double* [[TMP44]], i64 8
+; CHECK-NEXT:    [[TMP49:%.*]] = bitcast double* [[TMP48]] to <4 x double>*
+; CHECK-NEXT:    [[WIDE_LOAD9_EPIL:%.*]] = load <4 x double>, <4 x double>* [[TMP49]], align 8, [[TBAA3]], !alias.scope !7
+; CHECK-NEXT:    [[TMP50:%.*]] = getelementptr inbounds double, double* [[TMP44]], i64 12
+; CHECK-NEXT:    [[TMP51:%.*]] = bitcast double* [[TMP50]] to <4 x double>*
+; CHECK-NEXT:    [[WIDE_LOAD10_EPIL:%.*]] = load <4 x double>, <4 x double>* [[TMP51]], align 8, [[TBAA3]], !alias.scope !7
+; CHECK-NEXT:    [[TMP52:%.*]] = fmul fast <4 x double> [[WIDE_LOAD_EPIL]], [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[TMP53:%.*]] = fmul fast <4 x double> [[WIDE_LOAD8_EPIL]], [[BROADCAST_SPLAT12]]
+; CHECK-NEXT:    [[TMP54:%.*]] = fmul fast <4 x double> [[WIDE_LOAD9_EPIL]], [[BROADCAST_SPLAT14]]
+; CHECK-NEXT:    [[TMP55:%.*]] = fmul fast <4 x double> [[WIDE_LOAD10_EPIL]], [[BROADCAST_SPLAT16]]
+; CHECK-NEXT:    [[TMP56:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDEX_UNR]]
+; CHECK-NEXT:    [[TMP57:%.*]] = bitcast double* [[TMP56]] to <4 x double>*
+; CHECK-NEXT:    store <4 x double> [[TMP52]], <4 x double>* [[TMP57]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    [[TMP58:%.*]] = getelementptr inbounds double, double* [[TMP56]], i64 4
+; CHECK-NEXT:    [[TMP59:%.*]] = bitcast double* [[TMP58]] to <4 x double>*
+; CHECK-NEXT:    store <4 x double> [[TMP53]], <4 x double>* [[TMP59]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    [[TMP60:%.*]] = getelementptr inbounds double, double* [[TMP56]], i64 8
+; CHECK-NEXT:    [[TMP61:%.*]] = bitcast double* [[TMP60]] to <4 x double>*
+; CHECK-NEXT:    store <4 x double> [[TMP54]], <4 x double>* [[TMP61]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    [[TMP62:%.*]] = getelementptr inbounds double, double* [[TMP56]], i64 12
+; CHECK-NEXT:    [[TMP63:%.*]] = bitcast double* [[TMP62]] to <4 x double>*
+; CHECK-NEXT:    store <4 x double> [[TMP55]], <4 x double>* [[TMP63]], align 8, [[TBAA3]], !alias.scope !10, !noalias !7
+; CHECK-NEXT:    br label [[MIDDLE_BLOCK]]
 ; CHECK:       middle.block:
 ; CHECK-NEXT:    [[CMP_N:%.*]] = icmp eq i64 [[N_VEC]], [[WIDE_TRIP_COUNT]]
-; CHECK-NEXT:    br i1 [[CMP_N]], label [[FOR_END]], label [[FOR_BODY_PREHEADER]]
-; CHECK:       for.body.preheader:
-; CHECK-NEXT:    [[INDVARS_IV_PH:%.*]] = phi i64 [ 0, [[VECTOR_MEMCHECK]] ], [ 0, [[FOR_BODY_LR_PH]] ], [ [[N_VEC]], [[MIDDLE_BLOCK]] ]
-; CHECK-NEXT:    [[TMP34:%.*]] = xor i64 [[INDVARS_IV_PH]], -1
-; CHECK-NEXT:    [[TMP35:%.*]] = add nsw i64 [[TMP34]], [[WIDE_TRIP_COUNT]]
-; CHECK-NEXT:    [[XTRAITER8:%.*]] = and i64 [[WIDE_TRIP_COUNT]], 3
-; CHECK-NEXT:    [[LCMP_MOD9:%.*]] = icmp eq i64 [[XTRAITER8]], 0
-; CHECK-NEXT:    br i1 [[LCMP_MOD9]], label [[FOR_BODY_PROL_LOOPEXIT:%.*]], label [[FOR_BODY_PROL_PREHEADER:%.*]]
-; CHECK:       for.body.prol.preheader:
-; CHECK-NEXT:    [[TMP36:%.*]] = fdiv fast double 1.000000e+00, [[A]]
-; CHECK-NEXT:    br label [[FOR_BODY_PROL:%.*]]
+; CHECK-NEXT:    br i1 [[CMP_N]], label [[FOR_END]], label [[FOR_BODY_PREHEADER17]]
+; CHECK:       for.body.preheader17:
+; CHECK-NEXT:    [[INDVARS_IV_PH:%.*]] = phi i64 [ 0, [[VECTOR_MEMCHECK]] ], [ 0, [[FOR_BODY_PREHEADER]] ], [ [[N_VEC]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    [[TMP64:%.*]] = xor i64 [[INDVARS_IV_PH]], -1
+; CHECK-NEXT:    [[TMP65:%.*]] = add nsw i64 [[TMP64]], [[WIDE_TRIP_COUNT]]
+; CHECK-NEXT:    [[XTRAITER:%.*]] = and i64 [[WIDE_TRIP_COUNT]], 7
+; CHECK-NEXT:    [[LCMP_MOD_NOT:%.*]] = icmp eq i64 [[XTRAITER]], 0
+; CHECK-NEXT:    br i1 [[LCMP_MOD_NOT]], label [[FOR_BODY_PROL_LOOPEXIT:%.*]], label [[FOR_BODY_PROL:%.*]]
 ; CHECK:       for.body.prol:
-; CHECK-NEXT:    [[INDVARS_IV_PROL:%.*]] = phi i64 [ [[INDVARS_IV_NEXT_PROL:%.*]], [[FOR_BODY_PROL]] ], [ [[INDVARS_IV_PH]], [[FOR_BODY_PROL_PREHEADER]] ]
-; CHECK-NEXT:    [[PROL_ITER:%.*]] = phi i64 [ [[PROL_ITER_SUB:%.*]], [[FOR_BODY_PROL]] ], [ [[XTRAITER8]], [[FOR_BODY_PROL_PREHEADER]] ]
+; CHECK-NEXT:    [[INDVARS_IV_PROL:%.*]] = phi i64 [ [[INDVARS_IV_NEXT_PROL:%.*]], [[FOR_BODY_PROL]] ], [ [[INDVARS_IV_PH]], [[FOR_BODY_PREHEADER17]] ]
+; CHECK-NEXT:    [[PROL_ITER:%.*]] = phi i64 [ [[PROL_ITER_SUB:%.*]], [[FOR_BODY_PROL]] ], [ [[XTRAITER]], [[FOR_BODY_PREHEADER17]] ]
 ; CHECK-NEXT:    [[ARRAYIDX_PROL:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDVARS_IV_PROL]]
-; CHECK-NEXT:    [[T0_PROL:%.*]] = load double, double* [[ARRAYIDX_PROL]], align 8, !tbaa !3
-; CHECK-NEXT:    [[TMP37:%.*]] = fmul fast double [[T0_PROL]], [[TMP36]]
+; CHECK-NEXT:    [[T0_PROL:%.*]] = load double, double* [[ARRAYIDX_PROL]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[MUL_PROL:%.*]] = fmul fast double [[T0_PROL]], [[DIV]]
 ; CHECK-NEXT:    [[ARRAYIDX2_PROL:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDVARS_IV_PROL]]
-; CHECK-NEXT:    store double [[TMP37]], double* [[ARRAYIDX2_PROL]], align 8, !tbaa !3
+; CHECK-NEXT:    store double [[MUL_PROL]], double* [[ARRAYIDX2_PROL]], align 8, [[TBAA3]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT_PROL]] = add nuw nsw i64 [[INDVARS_IV_PROL]], 1
 ; CHECK-NEXT:    [[PROL_ITER_SUB]] = add i64 [[PROL_ITER]], -1
-; CHECK-NEXT:    [[PROL_ITER_CMP:%.*]] = icmp eq i64 [[PROL_ITER_SUB]], 0
-; CHECK-NEXT:    br i1 [[PROL_ITER_CMP]], label [[FOR_BODY_PROL_LOOPEXIT]], label [[FOR_BODY_PROL]], !llvm.loop !16
+; CHECK-NEXT:    [[PROL_ITER_CMP_NOT:%.*]] = icmp eq i64 [[PROL_ITER_SUB]], 0
+; CHECK-NEXT:    br i1 [[PROL_ITER_CMP_NOT]], label [[FOR_BODY_PROL_LOOPEXIT]], label [[FOR_BODY_PROL]], [[LOOP14:!llvm.loop !.*]]
 ; CHECK:       for.body.prol.loopexit:
-; CHECK-NEXT:    [[INDVARS_IV_UNR:%.*]] = phi i64 [ [[INDVARS_IV_PH]], [[FOR_BODY_PREHEADER]] ], [ [[INDVARS_IV_NEXT_PROL]], [[FOR_BODY_PROL]] ]
-; CHECK-NEXT:    [[TMP38:%.*]] = icmp ult i64 [[TMP35]], 3
-; CHECK-NEXT:    br i1 [[TMP38]], label [[FOR_END]], label [[FOR_BODY_PREHEADER_NEW:%.*]]
-; CHECK:       for.body.preheader.new:
-; CHECK-NEXT:    [[TMP39:%.*]] = fdiv fast double 1.000000e+00, [[A]]
-; CHECK-NEXT:    [[TMP40:%.*]] = fdiv fast double 1.000000e+00, [[A]]
-; CHECK-NEXT:    [[TMP41:%.*]] = fdiv fast double 1.000000e+00, [[A]]
-; CHECK-NEXT:    [[TMP42:%.*]] = fdiv fast double 1.000000e+00, [[A]]
-; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK-NEXT:    [[INDVARS_IV_UNR:%.*]] = phi i64 [ [[INDVARS_IV_PH]], [[FOR_BODY_PREHEADER17]] ], [ [[INDVARS_IV_NEXT_PROL]], [[FOR_BODY_PROL]] ]
+; CHECK-NEXT:    [[TMP66:%.*]] = icmp ult i64 [[TMP65]], 7
+; CHECK-NEXT:    br i1 [[TMP66]], label [[FOR_END]], label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
-; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[INDVARS_IV_UNR]], [[FOR_BODY_PREHEADER_NEW]] ], [ [[INDVARS_IV_NEXT_3:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[INDVARS_IV_NEXT_7:%.*]], [[FOR_BODY]] ], [ [[INDVARS_IV_UNR]], [[FOR_BODY_PROL_LOOPEXIT]] ]
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDVARS_IV]]
-; CHECK-NEXT:    [[T0:%.*]] = load double, double* [[ARRAYIDX]], align 8, !tbaa !3
-; CHECK-NEXT:    [[TMP43:%.*]] = fmul fast double [[T0]], [[TMP39]]
+; CHECK-NEXT:    [[T0:%.*]] = load double, double* [[ARRAYIDX]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[MUL:%.*]] = fmul fast double [[T0]], [[DIV]]
 ; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDVARS_IV]]
-; CHECK-NEXT:    store double [[TMP43]], double* [[ARRAYIDX2]], align 8, !tbaa !3
+; CHECK-NEXT:    store double [[MUL]], double* [[ARRAYIDX2]], align 8, [[TBAA3]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT:%.*]] = add nuw nsw i64 [[INDVARS_IV]], 1
 ; CHECK-NEXT:    [[ARRAYIDX_1:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDVARS_IV_NEXT]]
-; CHECK-NEXT:    [[T0_1:%.*]] = load double, double* [[ARRAYIDX_1]], align 8, !tbaa !3
-; CHECK-NEXT:    [[TMP44:%.*]] = fmul fast double [[T0_1]], [[TMP40]]
+; CHECK-NEXT:    [[T0_1:%.*]] = load double, double* [[ARRAYIDX_1]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[MUL_1:%.*]] = fmul fast double [[T0_1]], [[DIV]]
 ; CHECK-NEXT:    [[ARRAYIDX2_1:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDVARS_IV_NEXT]]
-; CHECK-NEXT:    store double [[TMP44]], double* [[ARRAYIDX2_1]], align 8, !tbaa !3
+; CHECK-NEXT:    store double [[MUL_1]], double* [[ARRAYIDX2_1]], align 8, [[TBAA3]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT_1:%.*]] = add nuw nsw i64 [[INDVARS_IV]], 2
 ; CHECK-NEXT:    [[ARRAYIDX_2:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDVARS_IV_NEXT_1]]
-; CHECK-NEXT:    [[T0_2:%.*]] = load double, double* [[ARRAYIDX_2]], align 8, !tbaa !3
-; CHECK-NEXT:    [[TMP45:%.*]] = fmul fast double [[T0_2]], [[TMP41]]
+; CHECK-NEXT:    [[T0_2:%.*]] = load double, double* [[ARRAYIDX_2]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[MUL_2:%.*]] = fmul fast double [[T0_2]], [[DIV]]
 ; CHECK-NEXT:    [[ARRAYIDX2_2:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDVARS_IV_NEXT_1]]
-; CHECK-NEXT:    store double [[TMP45]], double* [[ARRAYIDX2_2]], align 8, !tbaa !3
+; CHECK-NEXT:    store double [[MUL_2]], double* [[ARRAYIDX2_2]], align 8, [[TBAA3]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT_2:%.*]] = add nuw nsw i64 [[INDVARS_IV]], 3
 ; CHECK-NEXT:    [[ARRAYIDX_3:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDVARS_IV_NEXT_2]]
-; CHECK-NEXT:    [[T0_3:%.*]] = load double, double* [[ARRAYIDX_3]], align 8, !tbaa !3
-; CHECK-NEXT:    [[TMP46:%.*]] = fmul fast double [[T0_3]], [[TMP42]]
+; CHECK-NEXT:    [[T0_3:%.*]] = load double, double* [[ARRAYIDX_3]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[MUL_3:%.*]] = fmul fast double [[T0_3]], [[DIV]]
 ; CHECK-NEXT:    [[ARRAYIDX2_3:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDVARS_IV_NEXT_2]]
-; CHECK-NEXT:    store double [[TMP46]], double* [[ARRAYIDX2_3]], align 8, !tbaa !3
-; CHECK-NEXT:    [[INDVARS_IV_NEXT_3]] = add nuw nsw i64 [[INDVARS_IV]], 4
-; CHECK-NEXT:    [[EXITCOND_3:%.*]] = icmp eq i64 [[INDVARS_IV_NEXT_3]], [[WIDE_TRIP_COUNT]]
-; CHECK-NEXT:    br i1 [[EXITCOND_3]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop !17
+; CHECK-NEXT:    store double [[MUL_3]], double* [[ARRAYIDX2_3]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT_3:%.*]] = add nuw nsw i64 [[INDVARS_IV]], 4
+; CHECK-NEXT:    [[ARRAYIDX_4:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDVARS_IV_NEXT_3]]
+; CHECK-NEXT:    [[T0_4:%.*]] = load double, double* [[ARRAYIDX_4]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[MUL_4:%.*]] = fmul fast double [[T0_4]], [[DIV]]
+; CHECK-NEXT:    [[ARRAYIDX2_4:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDVARS_IV_NEXT_3]]
+; CHECK-NEXT:    store double [[MUL_4]], double* [[ARRAYIDX2_4]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT_4:%.*]] = add nuw nsw i64 [[INDVARS_IV]], 5
+; CHECK-NEXT:    [[ARRAYIDX_5:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDVARS_IV_NEXT_4]]
+; CHECK-NEXT:    [[T0_5:%.*]] = load double, double* [[ARRAYIDX_5]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[MUL_5:%.*]] = fmul fast double [[T0_5]], [[DIV]]
+; CHECK-NEXT:    [[ARRAYIDX2_5:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDVARS_IV_NEXT_4]]
+; CHECK-NEXT:    store double [[MUL_5]], double* [[ARRAYIDX2_5]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT_5:%.*]] = add nuw nsw i64 [[INDVARS_IV]], 6
+; CHECK-NEXT:    [[ARRAYIDX_6:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDVARS_IV_NEXT_5]]
+; CHECK-NEXT:    [[T0_6:%.*]] = load double, double* [[ARRAYIDX_6]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[MUL_6:%.*]] = fmul fast double [[T0_6]], [[DIV]]
+; CHECK-NEXT:    [[ARRAYIDX2_6:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDVARS_IV_NEXT_5]]
+; CHECK-NEXT:    store double [[MUL_6]], double* [[ARRAYIDX2_6]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT_6:%.*]] = add nuw nsw i64 [[INDVARS_IV]], 7
+; CHECK-NEXT:    [[ARRAYIDX_7:%.*]] = getelementptr inbounds double, double* [[Y]], i64 [[INDVARS_IV_NEXT_6]]
+; CHECK-NEXT:    [[T0_7:%.*]] = load double, double* [[ARRAYIDX_7]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[MUL_7:%.*]] = fmul fast double [[T0_7]], [[DIV]]
+; CHECK-NEXT:    [[ARRAYIDX2_7:%.*]] = getelementptr inbounds double, double* [[X]], i64 [[INDVARS_IV_NEXT_6]]
+; CHECK-NEXT:    store double [[MUL_7]], double* [[ARRAYIDX2_7]], align 8, [[TBAA3]]
+; CHECK-NEXT:    [[INDVARS_IV_NEXT_7]] = add nuw nsw i64 [[INDVARS_IV]], 8
+; CHECK-NEXT:    [[EXITCOND_NOT_7:%.*]] = icmp eq i64 [[INDVARS_IV_NEXT_7]], [[WIDE_TRIP_COUNT]]
+; CHECK-NEXT:    br i1 [[EXITCOND_NOT_7]], label [[FOR_END]], label [[FOR_BODY]], [[LOOP16:!llvm.loop !.*]]
 ; CHECK:       for.end:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %div = fdiv fast double 1.0, %a
   br label %for.cond
 
 for.cond: