[LV] Tail-loop Folding

[SjoerdMeijer]

Summary

This allows folding of the scalar epilogue loop (the tail) into the main
vectorised loop body when the loop is annotated with a "vector predicate"
metadata hint. To fold the tail, instructions need to be predicated (masked),
enabling/disabling lanes for the remainder iterations.

This depends on D64744 that introduces the llvm.loop.vectorize.predicate.enable
pragma and metadata node, and D64916 which is a refactoring to make tail
folding a more general concept.

[Meinersbur]

[serious] There is a LoopVectorizeHints class in LoopVectorizationLegality.cpp that should be used.

[serious] Documentation of llvm.loop.vectorize.predicate.enable is missing.

[Meinersbur]

Just just realized that docs for llvm.loop.vectorize.predicate.enable is part of D64744, which otherwise is a clang-only patch.

[SjoerdMeijer]

Hi Michael, thanks for taking a look again!

Just just realized that docs for llvm.loop.vectorize.predicate.enable is part of D64744, which otherwise is a clang-only patch.

Yep, indeed, so I assume that's all good.

There is a LoopVectorizeHints class in LoopVectorizationLegality.cpp that should be used.

Ah yes, thanks for the suggestion, I will start looking into this, and will move the pragma handling to some function in there.

[Meinersbur]

In D65197#1599082, @SjoerdMeijer wrote:

Just just realized that docs for llvm.loop.vectorize.predicate.enable is part of D64744, which otherwise is a clang-only patch.

Yep, indeed, so I assume that's all good.

Before we had the monoropo reviewers frequently asked to split patches into the LLVM and Clang part. With the monorepo, I am not sure the rule still needs to be followed. At least, I did not expect LLVM documentation in a clang patch, so sorry for the non-applicable comment.

[rengolin]

In D65197#1599102, @Meinersbur wrote:

Before we had the monoropo reviewers frequently asked to split patches into the LLVM and Clang part. With the monorepo, I am not sure the rule still needs to be followed. At least, I did not expect LLVM documentation in a clang patch, so sorry for the non-applicable comment.

It does because we're still committing to SVN. Once we enable write mode on the monorepo, that'll change.

[SjoerdMeijer]

Ha, that's funny, because before noticing these comments here, I was just doing a test commit (366904) with the github monorepo workflow.
With the discussion on the dev list that the transition date is near, and just following the public documentation in https://llvm.org/docs/GettingStarted.html, it really looks like I committed to the clang and llvm repo at the same time using the git llvm push script from a local git monorepo. I was of course aware of separating clang and llvm patches, but again, thought that this new workflow is fully accepted/supported.

Anyway, back to looking at LoopVectorizationLegality.cpp :-)

[hsaito]

We probably need to discuss whether vectorize_predicate(enable) should (or should not) implicitly turns on vectorize(enable) or not. I guess the current behavior is "does not", right? We don't have to discuss that in this review, but we still want to make a conscious decision one way or the other, or did I miss that discussion?

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

845

I think the nuance here is rather ScalarEpilogueNotNeededPredicatePragma. In other words, if scalar epilogue is needed for some other reason (but still okay to skip scalar epilogue execution when vector code executes), scalar epilogue can be emitted/utilized. Runtime vectorization legality check of all kinds fits in that profile. We shouldn't overload "predicated vector code" pragma with "don't emit scalar epilogue" meaning.

4765

-Os/-Oz message comes out from fall through. Not desired.

4784

How about

// Accept MaxVF if we don't have a tail at all.

and move the comment inside IF.

[SjoerdMeijer]

About:

We probably need to discuss whether vectorize_predicate(enable) should (or should not) implicitly turns on vectorize(enable) or not. I guess the current behavior is "does not", right? We don't have to discuss that in this review, but we still want to make a conscious decision one way or the other, or did I miss that discussion?

Nope, you're exactly right. We haven't discussed this yet, it had also crossed my mind, and we should discuss it. Your statement about the current behaviour is also right.

I will first look into addressing previous comments. My responses might be delayed due to an upcoming holiday, but finishing this is my highest priority.

[sdesmalen]

For SVE we found that there are sometimes benefits to using an unpredicated vector body plus a predicated tail. When the main vectorized loop-body is unpredicated, we know all lanes in the vector are executed and can produce more efficient set of instructions. The scalar tail can then still be vectorized using predication to mask off the inactive lanes, or depending on the cost of vectorizing the tail loop the compiler may want to choose not vectorizing the tail loop at all. It would be nice if your design allows for this use-case.
So maybe instead of having a boolean 'llvm.loop.vectorize.predicate.enable' you can make it into an enum, or perhaps rename the attribute to emphasises the difference so we can add this logic later?

llvm/lib/Analysis/LoopInfo.cpp

516

nit:

return Name.equals(S->getString()) &&
       mdconst::extract<ConstantInt>(MD->getOperand(1))->getZExtValue());

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

7344

nit: unnecessary whitespace.

[SjoerdMeijer]

Thanks for taking a look at this!

Some initial thoughts on this:

For SVE we found that there are sometimes benefits to using an unpredicated vector body plus a predicated tail. When the main vectorized loop-body is unpredicated, we know all lanes in the vector are executed and can produce more efficient set of instructions. The scalar tail can then still be vectorized using predication to mask off the inactive lanes, or depending on the cost of vectorizing the tail loop the compiler may want to choose not vectorizing the tail loop at all. It would be nice if your design allows for this use-case.
So maybe instead of having a boolean 'llvm.loop.vectorize.predicate.enable' you can make it into an enum, or perhaps rename the attribute to emphasises the difference so we can add this logic later?

In the current flow, the only use-case that we have so far, is that predicate.enable set by a pragma. As it is a pragma, like any other pragma, it is the user's responsibility whether this makes sense and is profitable, etc.

Another use case, is that predicate.enable is set by a loop vectorisation profitability analysis. Whether this is profitable or not, will indeed depend on the target (SVE, MVE, AVX, etc.), the core implementation, and different loop properties. So I can imagine that different target hooks will be required for this decision making, which can then result in setting predicate.enable. Thus, I don't think it excludes any use case, but in fact is the ground work for other use-cases.

[SjoerdMeijer]

I think I've addressed all comments, the main ones are:

• I've moved the loop hint handling to LoopVectorizationLegality.cpp
• I've renamed ScalarEpilogueNotNeededPredicatePragma
• and finally created a helper function to avoid some code duplication that has been bothering me for a while

[fhahn]

llvm/test/Transforms/LoopVectorize/tail_loop_folding.ll

2

It looks like assertions are not required for the test case.

6

If this test relies on the x86 cost model/x86 masked instructions, it should go into the subfolder I think.

[SjoerdMeijer]

I've moved the test case to the X86 subfolder and removed the ASSERT.

I obviously want to add some MVE tests too, but will do that later. The X86 cost model and masked instructions are a nice demonstrator :-)

[SjoerdMeijer]

ran clang-format.

[SjoerdMeijer]

Friendly ping :-)

[Meinersbur]

Looks fine to me, but see what the other reviewers say.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

859–869

[nit] this seems unrelated?

7254–7255

[nit] formatting-only change?

7449–7450

[nit] unrelated change?

[hsaito]

In D65197#1604940, @SjoerdMeijer wrote:

Friendly ping :-)

Looks like we are converging. One minor comment only.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

845

Thanks for addressing. May not be immediately effective, but should help if someone wants to move towards that direction.

4765

Thanks for taking care of it.

4784

Suggest moving this comment between the Lines 4806 and 4807.

[SjoerdMeijer]

Thanks for taking another look!

Feedback addressed, moved comment.

[SjoerdMeijer]

Sorry for asking, @hsaito , but I was just wondering and wanted to check if your last comment was in fact a LGTM with a minor nit.

[hsaito]

LGTM, pending the discussion about the exact meaning of the newly introduced "vector predicate" pragma (expect this to happen outside of this review). Please wait for another day to give others last minute opportunity to give feedback.

[SjoerdMeijer]

Many thanks for all your help and reviews!

I will start the discussion about the interaction between the vector predicate and vectorize pragmas as soon as I am back in the office. I will have a closer look first and try to form a better opinion, but my first thought at this moment is that enabling "vectorize_predicate" should simply imply "vectorize". As soon as I'm ready, I will upload a patch and perhaps a message to cfe dev.

llvm/include/llvm/Analysis/LoopInfo.h

public:
   /// Add llvm.loop.unroll.disable to this loop's loop id metadata.
   ///
   /// Remove existing unroll metadata and add unroll disable metadata to
   /// indicate the loop has already been unrolled.  This prevents a loop
   /// from being unrolled more than is directed by a pragma if the loop
   /// unrolling pass is run more than once (which it generally is).
   void setLoopAlreadyUnrolled();
 
+  /// Return true if the loop is annotated with pragma
+  /// llvm.loop.vectorize.predicate.enable, and false otherwise.
+  bool isAnnotatedVectorPredicate() const;
+
   void dump() const;
   void dumpVerbose() const;
 
   /// Return the debug location of the start of this loop.
   /// This looks for a BB terminating instruction with a known debug
   /// location by looking at the preheader and header blocks. If it
   /// cannot find a terminating instruction with location information,
   /// it returns an unknown location.

llvm/lib/Analysis/LoopInfo.cpp

void Loop::setLoopAlreadyUnrolled() {
   MDNode *DisableUnrollMD =
       MDNode::get(Context, MDString::get(Context, "llvm.loop.unroll.disable"));
   MDNode *LoopID = getLoopID();
   MDNode *NewLoopID = makePostTransformationMetadata(
       Context, LoopID, {"llvm.loop.unroll."}, {DisableUnrollMD});
   setLoopID(NewLoopID);
 }
 
+bool Loop::isAnnotatedVectorPredicate() const {
+  MDNode *LoopID = getLoopID();
+  if (!LoopID)
+    return false;
+
+  StringRef Name = "llvm.loop.vectorize.predicate.enable";
+  // First operand should refer to the loop id itself.
+  assert(LoopID->getNumOperands() > 0 && "requires at least one operand");
+  assert(LoopID->getOperand(0) == LoopID && "invalid loop id");
+
+  for (unsigned i = 1, e = LoopID->getNumOperands(); i < e; ++i) {
+    MDNode *MD = dyn_cast<MDNode>(LoopID->getOperand(i));
+    if (!MD)
+      continue;
+
+    MDString *S = dyn_cast<MDString>(MD->getOperand(0));
+    if (!S)
+      continue;
+
+    if (Name.equals(S->getString()) &&
+        mdconst::extract<ConstantInt>(MD->getOperand(1))->getZExtValue())
+      return true;
+    else
+      return false;
+  }
+  return false;
+}
+
 bool Loop::isAnnotatedParallel() const {
   MDNode *DesiredLoopIdMetadata = getLoopID();
 
   if (!DesiredLoopIdMetadata)
     return false;
 
   MDNode *ParallelAccesses =
       findOptionMDForLoop(this, "llvm.loop.parallel_accesses");

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

 
 namespace llvm {
 
 // Loop vectorization cost-model hints how the scalar epilogue loop should be
 // lowered.
 enum ScalarEpilogueLowering {
   CM_ScalarEpilogueAllowed,
   CM_ScalarEpilogueNotAllowedOptSize,
-  CM_ScalarEpilogueNotAllowedLowTripLoop
+  CM_ScalarEpilogueNotAllowedLowTripLoop,
+  CM_ScalarEpilogueNotAllowedPredicatePragma
 };
 
 /// LoopVectorizationCostModel - estimates the expected speedups due to
 /// vectorization.
 /// In many cases vectorization is not profitable. This can happen because of
 /// a number of reasons. In this class we mainly attempt to predict the
 /// expected speedup/slowdowns due to the supported instruction set. We use the
 /// TargetTransformInfo to query the different backends for the cost of
 /// different operations.
 class LoopVectorizationCostModel {
 public:
   LoopVectorizationCostModel(ScalarEpilogueLowering SEL, Loop *L,
                              PredicatedScalarEvolution &PSE,
                              LoopInfo *LI, LoopVectorizationLegality *Legal,
                              const TargetTransformInfo &TTI,
                              const TargetLibraryInfo *TLI, DemandedBits *DB,
                              AssumptionCache *AC,
                              OptimizationRemarkEmitter *ORE, const Function *F,
                              const LoopVectorizeHints *Hints,
                              InterleavedAccessInfo &IAI)
       : IsScalarEpilogueAllowed(SEL), TheLoop(L), PSE(PSE),
     LI(LI), Legal(Legal), TTI(TTI), TLI(TLI), DB(DB), AC(AC), ORE(ORE),
     TheFunction(F), Hints(Hints), InterleaveInfo(IAI) {}
 
   /// \return An upper bound for the vectorization factor, or None if
   /// vectorization and interleaving should be avoided up front.
   Optional<unsigned> computeMaxVF();
 
+  /// \return True if runtime checks are required for vectorization, and false
+  /// otherwise. 
+  bool runtimeChecksRequired();
+
   /// \return The most profitable vectorization factor and the cost of that VF.
   /// This method checks every power of two up to MaxVF. If UserVF is not ZERO
   /// then this vectorization factor will be selected if vectorization is
   /// possible.
   VectorizationFactor selectVectorizationFactor(unsigned MaxVF);
 
   /// Setup cost-based decisions for user vectorization factor.
   void selectUserVectorizationFactor(unsigned UserVF) {
for (auto &Induction : *Legal->getInductionVars()) {
     LLVM_DEBUG(dbgs() << "LV: Found uniform instruction: " << *Ind << "\n");
     LLVM_DEBUG(dbgs() << "LV: Found uniform instruction: " << *IndUpdate
                       << "\n");
   }
 
   Uniforms[VF].insert(Worklist.begin(), Worklist.end());
 }
 
-Optional<unsigned> LoopVectorizationCostModel::computeMaxVF() {
-  if (Legal->getRuntimePointerChecking()->Need && TTI.hasBranchDivergence()) {
-    // TODO: It may by useful to do since it's still likely to be dynamically
-    // uniform if the target can skip.
-    LLVM_DEBUG(
-        dbgs() << "LV: Not inserting runtime ptr check for divergent target");
-
-    ORE->emit(
-      createMissedAnalysis("CantVersionLoopWithDivergentTarget")
-      << "runtime pointer checks needed. Not enabled for divergent target");
-
-    return None;
-  }
-
-  unsigned TC = PSE.getSE()->getSmallConstantTripCount(TheLoop);
-  if (isScalarEpilogueAllowed())
-    return computeFeasibleMaxVF(TC);
-
-  LLVM_DEBUG(dbgs() << "LV: Not allowing scalar epilogue.\n" <<
-                       "LV: Performing code size checks.\n");
+bool LoopVectorizationCostModel::runtimeChecksRequired() {
+  LLVM_DEBUG(dbgs() << "LV: Performing code size checks.\n");
 
   if (Legal->getRuntimePointerChecking()->Need) {
     ORE->emit(createMissedAnalysis("CantVersionLoopWithOptForSize")
               << "runtime pointer checks needed. Enable vectorization of this "
                  "loop with '#pragma clang loop vectorize(enable)' when "
                  "compiling with -Os/-Oz");
     LLVM_DEBUG(
         dbgs()
         << "LV: Aborting. Runtime ptr check is required with -Os/-Oz.\n");
-    return None;
+    return true;
   }
 
   if (!PSE.getUnionPredicate().getPredicates().empty()) {
     ORE->emit(createMissedAnalysis("CantVersionLoopWithOptForSize")
               << "runtime SCEV checks needed. Enable vectorization of this "
                  "loop with '#pragma clang loop vectorize(enable)' when "
                  "compiling with -Os/-Oz");
     LLVM_DEBUG(
         dbgs()
         << "LV: Aborting. Runtime SCEV check is required with -Os/-Oz.\n");
-    return None;
+    return true;
   }
 
   // FIXME: Avoid specializing for stride==1 instead of bailing out.
   if (!Legal->getLAI()->getSymbolicStrides().empty()) {
     ORE->emit(createMissedAnalysis("CantVersionLoopWithOptForSize")
               << "runtime stride == 1 checks needed. Enable vectorization of "
                  "this loop with '#pragma clang loop vectorize(enable)' when "
                  "compiling with -Os/-Oz");
     LLVM_DEBUG(
         dbgs()
         << "LV: Aborting. Runtime stride check is required with -Os/-Oz.\n");
+    return true;
+  }
+
+  return false;
+}
+
+Optional<unsigned> LoopVectorizationCostModel::computeMaxVF() {
+  if (Legal->getRuntimePointerChecking()->Need && TTI.hasBranchDivergence()) {
+    // TODO: It may by useful to do since it's still likely to be dynamically
+    // uniform if the target can skip.
+    LLVM_DEBUG(
+        dbgs() << "LV: Not inserting runtime ptr check for divergent target");
+
+    ORE->emit(
+      createMissedAnalysis("CantVersionLoopWithDivergentTarget")
+      << "runtime pointer checks needed. Not enabled for divergent target");
+
     return None;
   }
 
-  // If we optimize the program for size, avoid creating the tail loop.
+  unsigned TC = PSE.getSE()->getSmallConstantTripCount(TheLoop);
   LLVM_DEBUG(dbgs() << "LV: Found trip count: " << TC << '\n');
-
   if (TC == 1) {
     ORE->emit(createMissedAnalysis("SingleIterationLoop")
               << "loop trip count is one, irrelevant for vectorization");
     LLVM_DEBUG(dbgs() << "LV: Aborting, single iteration (non) loop.\n");
     return None;
   }
 
-  // Record that scalar epilogue is not allowed.
+  switch (IsScalarEpilogueAllowed) {
+  default: return None;
+  case CM_ScalarEpilogueAllowed:
+    return computeFeasibleMaxVF(TC);
+  case CM_ScalarEpilogueNotAllowedPredicatePragma:
+    LLVM_DEBUG(dbgs() << "LV: vector predicate pragma found.\n"
+                      << "LV: creating predicated vector loop.\n");
+    break;
+  case CM_ScalarEpilogueNotAllowedLowTripLoop:
+    LLVM_DEBUG(dbgs() << "LV: Not allowing scalar epilogue due to low trip "
+                      << "count.\n");
+  case CM_ScalarEpilogueNotAllowedOptSize:
     LLVM_DEBUG(dbgs() << "LV: Not allowing scalar epilogue due to -Os/-Oz.\n");
+    // Bail if runtime checks are required, which are not good when optimising
+    // for size.
+    if (runtimeChecksRequired())
+      return None;
+    break;
+  }
+
+  // Now try the tail folding
 
-  // We don't create an epilogue when optimizing for size.
   // Invalidate interleave groups that require an epilogue if we can't mask
   // the interleave-group.
   if (!useMaskedInterleavedAccesses(TTI))
     InterleaveInfo.invalidateGroupsRequiringScalarEpilogue();
 
+  // Bail if we don't have a tail at all.
   unsigned MaxVF = computeFeasibleMaxVF(TC);
-
   if (TC > 0 && TC % MaxVF == 0) {
     LLVM_DEBUG(dbgs() << "LV: No tail will remain for any chosen VF.\n");
     return MaxVF;
   }
 
   // If we don't know the precise trip count, or if the trip count that we
   // found modulo the vectorization factor is not zero, try to fold the tail
   // by masking.
static bool processLoopInVPlanNativePath(
   Function *F = L->getHeader()->getParent();
   InterleavedAccessInfo IAI(PSE, L, DT, LI, LVL->getLAI());
 
   ScalarEpilogueLowering SEL = CM_ScalarEpilogueAllowed;
   if (Hints.getForce() != LoopVectorizeHints::FK_Enabled &&
       (F->hasOptSize() ||
        llvm::shouldOptimizeForSize(L->getHeader(), PSI, BFI)))
     SEL = CM_ScalarEpilogueNotAllowedOptSize;
+  else if (L->isAnnotatedVectorPredicate())
+    SEL = CM_ScalarEpilogueNotAllowedPredicatePragma;
 
   LoopVectorizationCostModel CM(SEL, L, PSE, LI, LVL, *TTI, TLI,
                                 DB, AC, ORE, F, &Hints, IAI);
   // Use the planner for outer loop vectorization.
   // TODO: CM is not used at this point inside the planner. Turn CM into an
   // optional argument if we don't need it in the future.
   LoopVectorizationPlanner LVP(L, LI, TLI, TTI, LVL, CM);
 
   // Get user vectorization factor.
   const unsigned UserVF = Hints.getWidth();
 
   // Plan how to best vectorize, return the best VF and its cost.
if (!LVL.canVectorize(EnableVPlanNativePath)) {
     LLVM_DEBUG(dbgs() << "LV: Not vectorizing: Cannot prove legality.\n");
     Hints.emitRemarkWithHints();
     return false;
   }
 
   // Check the function attributes and profiles to find out if this function
   // should be optimized for size.
   ScalarEpilogueLowering SEL = CM_ScalarEpilogueAllowed;
+
   if (Hints.getForce() != LoopVectorizeHints::FK_Enabled &&
       (F->hasOptSize() ||
        llvm::shouldOptimizeForSize(L->getHeader(), PSI, BFI)))
     SEL = CM_ScalarEpilogueNotAllowedOptSize;
+  else if (L->isAnnotatedVectorPredicate())
+    SEL = CM_ScalarEpilogueNotAllowedPredicatePragma;
 
   // Entrance to the VPlan-native vectorization path. Outer loops are processed
   // here. They may require CFG and instruction level transformations before
   // even evaluating whether vectorization is profitable. Since we cannot modify
   // the incoming IR, we need to build VPlan upfront in the vectorization
   // pipeline.
   if (!L->empty())
     return processLoopInVPlanNativePath(L, PSE, LI, DT, &LVL, TTI, TLI, DB, AC,
#endif /* NDEBUG */
 
   // Analyze interleaved memory accesses.
   if (UseInterleaved) {
     IAI.analyzeInterleaving(useMaskedInterleavedAccesses(*TTI));
   }
 
   // Use the cost model.
   LoopVectorizationCostModel CM(SEL, L, PSE, LI, &LVL, *TTI, TLI,
                                 DB, AC, ORE, F, &Hints, IAI);
   CM.collectValuesToIgnore();
 
   // Use the planner for vectorization.
   LoopVectorizationPlanner LVP(L, LI, TLI, TTI, &LVL, CM);
 
   // Get user vectorization factor.
   unsigned UserVF = Hints.getWidth();
 
   // Plan how to best vectorize, return the best VF and its cost.

llvm/test/Transforms/LoopVectorize/tail_loop_folding.ll

+; REQUIRES: asserts
+; RUN: opt < %s -loop-vectorize -S | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define dso_local void @tail_folding_enabled(i32* noalias nocapture %A, i32* noalias nocapture readonly %B, i32* noalias nocapture readonly %C) local_unnamed_addr #0 {
+; CHECK-LABEL: tail_folding_enabled(
+; CHECK:  vector.body:
+; CHECK:  %wide.masked.load = call <8 x i32> @llvm.masked.load.v8i32.p0v8i32(
+; CHECK:  %wide.masked.load1 = call <8 x i32> @llvm.masked.load.v8i32.p0v8i32(
+; CHECK:  %8 = add nsw <8 x i32> %wide.masked.load1, %wide.masked.load
+; CHECK:  call void @llvm.masked.store.v8i32.p0v8i32(
+; CHECK:  %index.next = add i64 %index, 8
+; CHECK:  %12 = icmp eq i64 %index.next, 432
+; CHECK:  br i1 %12, label %middle.block, label %vector.body, !llvm.loop !0
+
+entry:
+  br label %for.body
+
+for.cond.cleanup:
+  ret void
+
+for.body:
+  %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
+  %arrayidx = getelementptr inbounds i32, i32* %B, i64 %indvars.iv
+  %0 = load i32, i32* %arrayidx, align 4
+  %arrayidx2 = getelementptr inbounds i32, i32* %C, i64 %indvars.iv
+  %1 = load i32, i32* %arrayidx2, align 4
+  %add = add nsw i32 %1, %0
+  %arrayidx4 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
+  store i32 %add, i32* %arrayidx4, align 4
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+  %exitcond = icmp eq i64 %indvars.iv.next, 430
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body, !llvm.loop !6
+}
+
+define dso_local void @tail_folding_disabled(i32* noalias nocapture %A, i32* noalias nocapture readonly %B, i32* noalias nocapture readonly %C) local_unnamed_addr #0 {
+; CHECK-LABEL: tail_folding_disabled(
+; CHECK:      vector.body:
+; CHECK-NOT:  @llvm.masked.load.v8i32.p0v8i32(
+; CHECK-NOT:  @llvm.masked.store.v8i32.p0v8i32(
+; CHECK:      br i1 %44, label {{.*}}, label %vector.body
+entry:
+  br label %for.body
+
+for.cond.cleanup:
+  ret void
+
+for.body:
+  %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.body ]
+  %arrayidx = getelementptr inbounds i32, i32* %B, i64 %indvars.iv
+  %0 = load i32, i32* %arrayidx, align 4
+  %arrayidx2 = getelementptr inbounds i32, i32* %C, i64 %indvars.iv
+  %1 = load i32, i32* %arrayidx2, align 4
+  %add = add nsw i32 %1, %0
+  %arrayidx4 = getelementptr inbounds i32, i32* %A, i64 %indvars.iv
+  store i32 %add, i32* %arrayidx4, align 4
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+  %exitcond = icmp eq i64 %indvars.iv.next, 430
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body, !llvm.loop !10
+}
+
+; CHECK:      !0 = distinct !{!0, !1}
+; CHECK-NEXT: !1 = !{!"llvm.loop.isvectorized", i32 1}
+; CHECK-NEXT: !2 = distinct !{!2, !3, !1}
+; CHECK-NEXT: !3 = !{!"llvm.loop.unroll.runtime.disable"}
+; CHECK-NEXT: !4 = distinct !{!4, !1}
+; CHECK-NEXT: !5 = distinct !{!5, !3, !1}
+
+attributes #0 = { nounwind optsize uwtable "target-cpu"="core-avx2" "target-features"="+avx,+avx2" }
+
+!6 = distinct !{!6, !7, !8}
+!7 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+
+!10 = distinct !{!10, !11, !12}
+!11 = !{!"llvm.loop.vectorize.predicate.enable", i1 false}
+!12 = !{!"llvm.loop.vectorize.enable", i1 true}