[LV] Allow vectorization of hot short trip count loops with epilog

[ebrevnov]

Summary

Currently we reject to vectorize loops with epilog if trip count less than 16 (as defined by TinyTripCountVectorThreshold) and loop is cold according to profile. Thus in absence of profile summary information we will never vectorize such loops even if cost model can prove profitability. With this change we try to improve the situation by using block execution count information (if available) to judge if loop is hot relatively to function entry.

Please note that I moved related code to getScalarEpilogueLowering. That not only improves maintainability but fixes potential issue of redefining decision made by getScalarEpilogueLowering by short trip count heuristic.

Ideally we should be using loop size and expected perf gain from cost model in addition to loop hotness to make the decision but that would be much bigger change and can be a follow up step.

Note that this change depends on https://reviews.llvm.org/D67690

[ebrevnov]

Minor update

[ebrevnov]

Minor changes in tests.

[ebrevnov]

llvm/include/llvm/Transforms/Utils/SizeOpts.h

78–81

This change is motivated by use case in getScalarEpilogueLowering.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

943

just to be consistent with the next line :-)

[ebrevnov]

Test update

[ebrevnov]

Minor test update

[ebrevnov]

ping

[reames]

Heuristics looks pretty reasonable to me, but I'd really like to have someone more familiar with how all the existing hueristics interact give the actual LGTM.

llvm/include/llvm/Transforms/Utils/SizeOpts.h

77

Please update comments to indicate what None value means. Your verbal explanation was clear, just make sure the code reflects that.

After that, please separate and land the refactoring change involving these two methods. Then rebase.

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

7441

Explicit type this please for readability.

[ebrevnov]

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

7465

Some rational for the chosen heuristic. In general if nobody actually asked to optimize for size it seems reasonable to relay on cost model to decide if vectorization is profitable or not even for short trip count loops. If we still want to have some balance between code bloat and performance we should decide based on potential gain and loop size for all loops. Even though the described approach looks simple and reasonable in theory it most likely will have big implications on existing apps. That's why I decided take more conservative approach and give a chance for hot loops to be vectorized.

[ebrevnov]

Rebase.

[ebrevnov]

Rebase. Attempt2.

[Ayal]

Ideally we should be using loop size and expected perf gain from cost model in addition to loop hotness to make the decision but that would be much bigger change and can be a follow up step.

The original motivation to vectorize loops of tiny trip counts as if under opt-for-size, is not to save code size, nor is it related to how hot the loop is. It stems from the fact that LV's cost model currently focuses on the body of the vector loop only, disregarding overheads associated with runtime guards and code outside this body, under the assumption that the execution of the vector loop will dominate. If the trip count is tiny, this assumption no longer holds, and the overheads may outweigh the benefits, w/o LV's cost-model noticing, leading to slowdowns (which become more severe as the loop gets hotter). Under opt-for-size LV generates only the vector loop, with no runtime guards nor scalar epilog, so this assumption holds.

In order to vectorize loops of tiny trip count with a scalar epilogue, the cost model should be enhanced to account for both plus associated overheads.

Regarding hotness, that could help the cost model in general to compare the expected performance with that of the original loop, but not sure how in this case where the trip count is a known constant.

[ebrevnov]

I'll start a separate review since the approach should be completely different.

llvm/include/llvm/CodeGen/MachineSizeOpts.h

 namespace llvm {
 
 class ProfileSummaryInfo;
 class MachineBasicBlock;
 class MachineBlockFrequencyInfo;
 class MachineFunction;
 
 /// Returns true if machine function \p MF is suggested to be size-optimized
-/// base on the profile.
-bool shouldOptimizeForSize(const MachineFunction *MF, ProfileSummaryInfo *PSI,
+/// base on the profile, false if not. Returns None if there is no info the
+/// profile.
+Optional<bool> shouldOptimizeForSize(const MachineFunction *MF,
+                                     ProfileSummaryInfo *PSI,
                                      const MachineBlockFrequencyInfo *BFI);
 /// Returns true if machine basic block \p MBB is suggested to be size-optimized
-/// base on the profile.
-bool shouldOptimizeForSize(const MachineBasicBlock *MBB,
+/// base on the profile, false if not. Returns None if there is no info the
+/// profile.
+Optional<bool> shouldOptimizeForSize(const MachineBasicBlock *MBB,
                                      ProfileSummaryInfo *PSI,
                                      const MachineBlockFrequencyInfo *MBFI);
 
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_MACHINE_SIZEOPTS_H

llvm/include/llvm/Transforms/Utils/SizeOpts.h

 
 namespace llvm {
 
 class BasicBlock;
 class BlockFrequencyInfo;
 class Function;
 class ProfileSummaryInfo;
 
-template<typename AdapterT, typename FuncT, typename BFIT>
-bool shouldFuncOptimizeForSizeImpl(const FuncT *F, ProfileSummaryInfo *PSI,
+template <typename AdapterT, typename FuncT, typename BFIT>
+Optional<bool> shouldFuncOptimizeForSizeImpl(const FuncT *F,
+                                             ProfileSummaryInfo *PSI,
                                              BFIT *BFI) {
   assert(F);
   if (!PSI || !BFI || !PSI->hasProfileSummary())
-    return false;
+    return None;
   if (ForcePGSO)
     return true;
   if (!EnablePGSO)
-    return false;
+    return None;
   if (PGSOColdCodeOnly ||
       (PGSOLargeWorkingSetSizeOnly && !PSI->hasLargeWorkingSetSize())) {
     // Even if the working set size isn't large, size-optimize cold code.
     return AdapterT::isFunctionColdInCallGraph(F, PSI, *BFI);
   }
   return !AdapterT::isFunctionHotInCallGraphNthPercentile(
       PSI->hasSampleProfile() ? PgsoCutoffSampleProf : PgsoCutoffInstrProf,
       F, PSI, *BFI);
 }
 
-template<typename AdapterT, typename BlockT, typename BFIT>
-bool shouldOptimizeForSizeImpl(const BlockT *BB, ProfileSummaryInfo *PSI,
-                               BFIT *BFI) {
+template <typename AdapterT, typename BlockT, typename BFIT>
+Optional<bool> shouldOptimizeForSizeImpl(const BlockT *BB,
+                                         ProfileSummaryInfo *PSI, BFIT *BFI) {
   assert(BB);
   if (!PSI || !BFI || !PSI->hasProfileSummary())
-    return false;
+    return None;
   if (ForcePGSO)
     return true;
   if (!EnablePGSO)
-    return false;
+    return None;
   if (PGSOColdCodeOnly ||
       (PGSOLargeWorkingSetSizeOnly && !PSI->hasLargeWorkingSetSize())) {
     // Even if the working set size isn't large, size-optimize cold code.
     return AdapterT::isColdBlock(BB, PSI, BFI);
   }
   return !AdapterT::isHotBlockNthPercentile(
       PSI->hasSampleProfile() ? PgsoCutoffSampleProf : PgsoCutoffInstrProf,
       BB, PSI, BFI);
 }
 
 /// Returns true if function \p F is suggested to be size-optimized base on the
-/// profile.
-bool shouldOptimizeForSize(const Function *F, ProfileSummaryInfo *PSI,
+/// profile, false if not. Returns None if there is no info the profile.
+Optional<bool> shouldOptimizeForSize(const Function *F, ProfileSummaryInfo *PSI,
                                      BlockFrequencyInfo *BFI);
 
 /// Returns true if basic block \p BB is suggested to be size-optimized base
-/// on the profile.
-bool shouldOptimizeForSize(const BasicBlock *BB, ProfileSummaryInfo *PSI,
+/// on the profile, false if not. Returns None if there is no info the profile.
+Optional<bool> shouldOptimizeForSize(const BasicBlock *BB,
+                                     ProfileSummaryInfo *PSI,
                                      BlockFrequencyInfo *BFI);
-
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_UTILS_SIZEOPTS_H

llvm/lib/CodeGen/MachineSizeOpts.cpp

static bool isHotBlockNthPercentile(int CutOff,
                                       ProfileSummaryInfo *PSI,
                                       const MachineBlockFrequencyInfo *MBFI) {
     return machine_size_opts_detail::isHotBlockNthPercentile(
         CutOff, MBB, PSI, MBFI);
   }
 };
 } // end anonymous namespace
 
-bool llvm::shouldOptimizeForSize(const MachineFunction *MF,
-                                 ProfileSummaryInfo *PSI,
-                                 const MachineBlockFrequencyInfo *MBFI) {
-  return shouldFuncOptimizeForSizeImpl<MachineBasicBlockBFIAdapter>(
-      MF, PSI, MBFI);
+Optional<bool>
+llvm::shouldOptimizeForSize(const MachineFunction *MF, ProfileSummaryInfo *PSI,
+  return shouldFuncOptimizeForSizeImpl<MachineBasicBlockBFIAdapter>(MF, PSI,
+                                                                    MBFI);
 }
 
-bool llvm::shouldOptimizeForSize(const MachineBasicBlock *MBB,
-                                 ProfileSummaryInfo *PSI,
-                                 const MachineBlockFrequencyInfo *MBFI) {
-  return shouldOptimizeForSizeImpl<MachineBasicBlockBFIAdapter>(
-      MBB, PSI, MBFI);
+Optional<bool>
+llvm::shouldOptimizeForSize(const MachineBasicBlock *MBB,
+  return shouldOptimizeForSizeImpl<MachineBasicBlockBFIAdapter>(MBB, PSI, MBFI);
 }

llvm/lib/CodeGen/TargetLoweringBase.cpp

bool TargetLoweringBase::isSuitableForJumpTable(const SwitchInst *SI,
                                                 BlockFrequencyInfo *BFI) const {
   // FIXME: This function check the maximum table size and density, but the
   // minimum size is not checked. It would be nice if the minimum size is
   // also combined within this function. Currently, the minimum size check is
   // performed in findJumpTable() in SelectionDAGBuiler and
   // getEstimatedNumberOfCaseClusters() in BasicTTIImpl.
   const bool OptForSize =
       SI->getParent()->getParent()->hasOptSize() ||
-      llvm::shouldOptimizeForSize(SI->getParent(), PSI, BFI);
+      llvm::shouldOptimizeForSize(SI->getParent(), PSI, BFI).getValueOr(false);
   const unsigned MinDensity = getMinimumJumpTableDensity(OptForSize);
   const unsigned MaxJumpTableSize = getMaximumJumpTableSize();
 
   // Check whether the number of cases is small enough and
   // the range is dense enough for a jump table.
   return (OptForSize || Range <= MaxJumpTableSize) &&
          (NumCases * 100 >= Range * MinDensity);
 }

llvm/lib/Transforms/Scalar/ConstantHoisting.cpp

 // which does not do all the offset calculations.
 unsigned
 ConstantHoistingPass::maximizeConstantsInRange(ConstCandVecType::iterator S,
                                            ConstCandVecType::iterator E,
                                            ConstCandVecType::iterator &MaxCostItr) {
   unsigned NumUses = 0;
 
   bool OptForSize = Entry->getParent()->hasOptSize() ||
-                    llvm::shouldOptimizeForSize(Entry->getParent(), PSI, BFI);
-  if (!OptForSize || std::distance(S,E) > 100) {
+                    llvm::shouldOptimizeForSize(Entry->getParent(), PSI, BFI)
+                        .getValueOr(false);
+  if (!OptForSize || std::distance(S, E) > 100) {
     for (auto ConstCand = S; ConstCand != E; ++ConstCand) {
       NumUses += ConstCand->Uses.size();
       if (ConstCand->CumulativeCost > MaxCostItr->CumulativeCost)
         MaxCostItr = ConstCand;
     }
     return NumUses;
   }
 

llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp

if (!Checks.empty() || !LAI.getPSE().getUnionPredicate().isAlwaysTrue()) {
       if (LAI.hasConvergentOp()) {
         LLVM_DEBUG(dbgs() << "Versioning is needed but not allowed with "
                              "convergent calls\n");
         return false;
       }
 
       auto *HeaderBB = L->getHeader();
       auto *F = HeaderBB->getParent();
-      bool OptForSize = F->hasOptSize() ||
-                        llvm::shouldOptimizeForSize(HeaderBB, PSI, BFI);
+      bool OptForSize =
+          F->hasOptSize() ||
+          llvm::shouldOptimizeForSize(HeaderBB, PSI, BFI).getValueOr(false);
       if (OptForSize) {
         LLVM_DEBUG(
             dbgs() << "Versioning is needed but not allowed when optimizing "
                       "for size.\n");
         return false;
       }
 
       if (!L->isLoopSimplifyForm()) {

llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp

TargetTransformInfo::UnrollingPreferences llvm::gatherUnrollingPreferences(
   UP.UnrollAndJam = false;
   UP.PeelProfiledIterations = true;
   UP.UnrollAndJamInnerLoopThreshold = 60;
 
   // Override with any target specific settings
   TTI.getUnrollingPreferences(L, SE, UP);
 
   // Apply size attributes
-  bool OptForSize = L->getHeader()->getParent()->hasOptSize() ||
-                    llvm::shouldOptimizeForSize(L->getHeader(), PSI, BFI);
+  bool OptForSize =
+      L->getHeader()->getParent()->hasOptSize() ||
+      llvm::shouldOptimizeForSize(L->getHeader(), PSI, BFI).getValueOr(false);
   if (OptForSize) {
     UP.Threshold = UP.OptSizeThreshold;
     UP.PartialThreshold = UP.PartialOptSizeThreshold;
     UP.MaxPercentThresholdBoost = 100;
   }
 
   // Apply any user values specified by cl::opt
   if (UnrollThreshold.getNumOccurrences() > 0)

llvm/lib/Transforms/Utils/SimplifyLibCalls.cpp

Value *LibCallSimplifier::optimizeFWrite(CallInst *CI, IRBuilder<> &B) {
   return nullptr;
 }
 
 Value *LibCallSimplifier::optimizeFPuts(CallInst *CI, IRBuilder<> &B) {
   optimizeErrorReporting(CI, B, 1);
 
   // Don't rewrite fputs to fwrite when optimising for size because fwrite
   // requires more arguments and thus extra MOVs are required.
-  bool OptForSize = CI->getFunction()->hasOptSize() ||
-                    llvm::shouldOptimizeForSize(CI->getParent(), PSI, BFI);
+  bool OptForSize =
+      CI->getFunction()->hasOptSize() ||
+      llvm::shouldOptimizeForSize(CI->getParent(), PSI, BFI).getValueOr(false);
   if (OptForSize)
     return nullptr;
 
   // Check if has any use
   if (!CI->use_empty()) {
     if (isLocallyOpenedFile(CI->getArgOperand(1), CI, B, TLI))
       return emitFPutSUnlocked(CI->getArgOperand(0), CI->getArgOperand(1), B,
                                TLI);

llvm/lib/Transforms/Utils/SizeOpts.cpp

static bool isHotBlockNthPercentile(int CutOff,
                                       const BasicBlock *BB,
                                       ProfileSummaryInfo *PSI,
                                       BlockFrequencyInfo *BFI) {
     return PSI->isHotBlockNthPercentile(CutOff, BB, BFI);
   }
 };
 } // end anonymous namespace
 
-bool llvm::shouldOptimizeForSize(const Function *F, ProfileSummaryInfo *PSI,
+Optional<bool> llvm::shouldOptimizeForSize(const Function *F,
+                                           ProfileSummaryInfo *PSI,
                                            BlockFrequencyInfo *BFI) {
   return shouldFuncOptimizeForSizeImpl<BasicBlockBFIAdapter>(F, PSI, BFI);
 }
 
-bool llvm::shouldOptimizeForSize(const BasicBlock *BB, ProfileSummaryInfo *PSI,
+Optional<bool> llvm::shouldOptimizeForSize(const BasicBlock *BB,
+                                           ProfileSummaryInfo *PSI,
                                            BlockFrequencyInfo *BFI) {
   return shouldOptimizeForSizeImpl<BasicBlockBFIAdapter>(BB, PSI, BFI);
 }

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

 
 cl::opt<bool> llvm::EnableLoopInterleaving(
     "interleave-loops", cl::init(true), cl::Hidden,
     cl::desc("Enable loop interleaving in Loop vectorization passes"));
 cl::opt<bool> llvm::EnableLoopVectorization(
     "vectorize-loops", cl::init(true), cl::Hidden,
     cl::desc("Run the Loop vectorization passes"));
 
+static cl::opt<unsigned> LocalHotnessThreshold(
+    "local-hotness-threshold", cl::init(500), cl::Hidden,
+    cl::desc(
+        "In cases when there is no info on block hotness available from module "
+        "profile we define \"local hotness\" as a ratio of the block to "
+        "function entry execution counts. If the ration is greater than the "
+        "threshold defined by this parameter the block is said to be locally "
+        "hot."));
+
 /// A helper function for converting Scalar types to vector types.
 /// If the incoming type is void, we return void. If the VF is 1, we return
 /// the scalar type.
 static Type *ToVectorTy(Type *Scalar, unsigned VF) {
   if (Scalar->isVoidTy() || VF == 1)
     return Scalar;
   return VectorType::get(Scalar, VF);
 }

 enum ScalarEpilogueLowering {
 
   // The default: allowing scalar epilogues.
   CM_ScalarEpilogueAllowed,
 
   // Vectorization with OptForSize: don't allow epilogues.
   CM_ScalarEpilogueNotAllowedOptSize,
 
   // A special case of vectorisation with OptForSize: loops with a very small
   // trip count are considered for vectorization under OptForSize, thereby
   // making sure the cost of their loop body is dominant, free of runtime
   // guards and scalar iteration overheads.
   CM_ScalarEpilogueNotAllowedLowTripLoop,
 
   // Loop hint predicate indicating an epilogue is undesired.
   CM_ScalarEpilogueNotNeededUsePredicate
 };

 static ScalarEpilogueLowering
 getScalarEpilogueLowering(Function *F, Loop *L, LoopVectorizeHints &Hints,
                           ProfileSummaryInfo *PSI, BlockFrequencyInfo *BFI,
                           TargetTransformInfo *TTI, TargetLibraryInfo *TLI,
                           AssumptionCache *AC, LoopInfo *LI,
                           ScalarEvolution *SE, DominatorTree *DT,
                           const LoopAccessInfo *LAI) {
   ScalarEpilogueLowering SEL = CM_ScalarEpilogueAllowed;
+  Optional<bool> IsColdByProfile =
+      llvm::shouldOptimizeForSize(L->getHeader(), PSI, BFI);
   if (Hints.getForce() != LoopVectorizeHints::FK_Enabled &&
-      (F->hasOptSize() ||
-       llvm::shouldOptimizeForSize(L->getHeader(), PSI, BFI)))
+      (F->hasOptSize() || IsColdByProfile.getValueOr(false)))
     SEL = CM_ScalarEpilogueNotAllowedOptSize;
   else if (PreferPredicateOverEpilog ||
            Hints.getPredicate() == LoopVectorizeHints::FK_Enabled ||
            (TTI->preferPredicateOverEpilogue(L, LI, *SE, *AC, TLI, DT, LAI) &&
             Hints.getPredicate() != LoopVectorizeHints::FK_Disabled))
     SEL = CM_ScalarEpilogueNotNeededUsePredicate;
+  else {
+    auto ExpectedTC = getSmallBestKnownTC(*SE, L);
+    // Check the loop for a trip count threshold: vectorize loops with a tiny
+    // trip count by optimizing for size, to minimize overheads.
+    if (ExpectedTC && *ExpectedTC < TinyTripCountVectorThreshold) {
+      // Even short trip count loops may be hot (part of hot region).
+      // In absence of profile summary estimate loop hotness relative to
+      // function entry using execution frequency information.
+      if (!IsColdByProfile && LoopVectorizeWithBlockFrequency && BFI) {
+        Optional<uint64_t> LoopCount =
+            BFI->getBlockProfileCount(L->getHeader(), true);
+        Optional<uint64_t> FunctionCount =
+            BFI->getBlockProfileCount(&F->getEntryBlock(), true);
+        if (LoopCount && FunctionCount &&
+            (*LoopCount > *FunctionCount * LocalHotnessThreshold)) {
+          LLVM_DEBUG(dbgs() << "Allow epilog for short trip count loop due to "
+                               "hotness considerations.");
+          return CM_ScalarEpilogueAllowed;
+        }
+      }
+
+      LLVM_DEBUG(dbgs() << "LV: Found a loop with a very small trip count. "
+                        << "This loop is worth vectorizing only if no scalar "
+                        << "iteration overheads are incurred.");
+
+      if (Hints.getForce() == LoopVectorizeHints::FK_Enabled)
+        LLVM_DEBUG(dbgs() << " But vectorizing was explicitly forced.\n");
+      else {
+        LLVM_DEBUG(dbgs() << "\n");
+        SEL = CM_ScalarEpilogueNotAllowedLowTripLoop;
+      }
+    }
+  }
 
   return SEL;
 }
 
 // Process the loop in the VPlan-native vectorization path. This path builds
 // VPlan upfront in the vectorization pipeline, which allows to apply
 // VPlan-to-VPlan transformations from the very beginning without modifying the
 // input LLVM IR.
#endif /* NDEBUG */
   // the incoming IR, we need to build VPlan upfront in the vectorization
   // pipeline.
   if (!L->empty())
     return processLoopInVPlanNativePath(L, PSE, LI, DT, &LVL, TTI, TLI, DB, AC,
                                         ORE, BFI, PSI, Hints);
 
   assert(L->empty() && "Inner loop expected.");
 
-  // Check the loop for a trip count threshold: vectorize loops with a tiny trip
-  // count by optimizing for size, to minimize overheads.
-  auto ExpectedTC = getSmallBestKnownTC(*SE, L);
-  if (ExpectedTC && *ExpectedTC < TinyTripCountVectorThreshold) {
-    LLVM_DEBUG(dbgs() << "LV: Found a loop with a very small trip count. "
-                      << "This loop is worth vectorizing only if no scalar "
-                      << "iteration overheads are incurred.");
-    if (Hints.getForce() == LoopVectorizeHints::FK_Enabled)
-      LLVM_DEBUG(dbgs() << " But vectorizing was explicitly forced.\n");
-    else {
-      LLVM_DEBUG(dbgs() << "\n");
-      SEL = CM_ScalarEpilogueNotAllowedLowTripLoop;
-    }
-  }
-
   // Check the function attributes to see if implicit floats are allowed.
   // FIXME: This check doesn't seem possibly correct -- what if the loop is
   // an integer loop and the vector instructions selected are purely integer
   // vector instructions?
   if (F->hasFnAttribute(Attribute::NoImplicitFloat)) {
     reportVectorizationFailure(
         "Can't vectorize when the NoImplicitFloat attribute is used",
         "loop not vectorized due to NoImplicitFloat attribute",

llvm/test/Transforms/LoopVectorize/hot_short_tc_loop.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -passes="print<block-freq>,loop-vectorize" -S < %s 2>&1 |  FileCheck %s
+
+; Check vectorization of hot short trip count with epilog. In this case inner
+; loop trip count is not constant and its value is estimated by profile.
+
+; ModuleID = 'test.cpp'
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+@a = dso_local global [5 x i32] zeroinitializer, align 16
+@b = dso_local global [5 x i32] zeroinitializer, align 16
+
+; Function Attrs: uwtable
+define dso_local void @_Z3fooi(i32 %M) local_unnamed_addr #0 !prof !11 {
+; CHECK:    [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP15:%.*]]
+; CHECK:    [[TMP18:%.*]] = mul nsw <4 x i32> [[WIDE_LOAD]], [[VEC_IND6:%.*]]
+; CHECK:    [[WIDE_LOAD10:%.*]] = load <4 x i32>, <4 x i32>* [[TMP23:%.*]]
+; CHECK:    [[TMP26:%.*]] = add nsw <4 x i32> [[WIDE_LOAD10]], [[TMP18]]
+; CHECK:    store <4 x i32> [[TMP26]], <4 x i32>* [[TMP28:%.*]]
+;
+entry:
+  %a = alloca [5 x i32], align 16
+  %b = alloca [5 x i32], align 16
+  %0 = bitcast [5 x i32]* %a to i8*
+  call void @llvm.lifetime.start.p0i8(i64 20, i8* nonnull %0) #3
+  %1 = bitcast [5 x i32]* %b to i8*
+  call void @llvm.lifetime.start.p0i8(i64 20, i8* nonnull %1) #3
+  %arraydecay = getelementptr inbounds [5 x i32], [5 x i32]* %a, i64 0, i64 0
+  br label %for.body.us.preheader
+
+for.body.us.preheader:                            ; preds = %entry
+  %wide.trip.count = zext i32 %M to i64
+  br label %for.body.us
+
+for.body.us:                                      ; preds = %for.cond1.for.cond.cleanup3_crit_edge.us, %for.body.us.preheader
+  %j.019.us = phi i32 [ %inc8.us, %for.cond1.for.cond.cleanup3_crit_edge.us ], [ 0, %for.body.us.preheader ]
+  call void @_Z3barPi(i32* nonnull %arraydecay)
+  br label %for.body4.us
+
+for.body4.us:                                     ; preds = %for.body4.us, %for.body.us
+  %indvars.iv = phi i64 [ 0, %for.body.us ], [ %indvars.iv.next, %for.body4.us ]
+  %arrayidx.us = getelementptr inbounds [5 x i32], [5 x i32]* %b, i64 0, i64 %indvars.iv
+  %2 = load i32, i32* %arrayidx.us, align 4, !tbaa !2
+  %3 = trunc i64 %indvars.iv to i32
+  %mul.us = mul nsw i32 %2, %3
+  %arrayidx6.us = getelementptr inbounds [5 x i32], [5 x i32]* %a, i64 0, i64 %indvars.iv
+  %4 = load i32, i32* %arrayidx6.us, align 4, !tbaa !2
+  %add.us = add nsw i32 %4, %mul.us
+  store i32 %add.us, i32* %arrayidx6.us, align 4, !tbaa !2
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+  %exitcond = icmp eq i64 %indvars.iv.next, %wide.trip.count
+  br i1 %exitcond, label %for.cond1.for.cond.cleanup3_crit_edge.us, label %for.body4.us, !prof !10
+
+for.cond1.for.cond.cleanup3_crit_edge.us:         ; preds = %for.body4.us
+  %inc8.us = add nuw nsw i32 %j.019.us, 1
+  %exitcond21 = icmp eq i32 %inc8.us, 20
+  br i1 %exitcond21, label %for.cond.cleanup.loopexit, label %for.body.us, !prof !12
+
+for.cond.cleanup.loopexit:                        ; preds = %for.cond1.for.cond.cleanup3_crit_edge.us
+  br label %for.cond.cleanup
+
+for.cond.cleanup.loopexit24:                      ; preds = %for.body
+  br label %for.cond.cleanup
+
+for.cond.cleanup:                                 ; preds = %for.cond.cleanup.loopexit24, %for.cond.cleanup.loopexit
+  call void @llvm.lifetime.end.p0i8(i64 20, i8* nonnull %1) #3
+  call void @llvm.lifetime.end.p0i8(i64 20, i8* nonnull %0) #3
+  ret void
+}
+
+; Check vectorization of hot short trip count with epilog. In this case inner
+; loop trip count is known constant value.
+
+; Function Attrs: uwtable
+define dso_local void @_Z3fooi2() local_unnamed_addr #0 !prof !11 {
+; CHECK:    [[WIDE_LOAD:%.*]] = load <4 x i32>, <4 x i32>* [[TMP15:%.*]]
+; CHECK:    [[TMP18:%.*]] = mul nsw <4 x i32> [[WIDE_LOAD]], [[VEC_IND6:%.*]]
+; CHECK:    [[WIDE_LOAD10:%.*]] = load <4 x i32>, <4 x i32>* [[TMP23:%.*]]
+; CHECK:    [[TMP26:%.*]] = add nsw <4 x i32> [[WIDE_LOAD10]], [[TMP18]]
+; CHECK:    store <4 x i32> [[TMP26]], <4 x i32>* [[TMP28:%.*]]
+;
+entry:
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.cond.cleanup3
+  ret void
+
+for.body:                                         ; preds = %entry, %for.cond.cleanup3
+  %j.018 = phi i32 [ 0, %entry ], [ %inc8, %for.cond.cleanup3 ]
+  tail call void @_Z3barPi(i32* getelementptr inbounds ([5 x i32], [5 x i32]* @a, i64 0, i64 0))
+  br label %for.body4
+
+for.cond.cleanup3:                                ; preds = %for.body4
+  %inc8 = add nuw nsw i32 %j.018, 1
+  %cmp = icmp ult i32 %inc8, 1000
+  br i1 %cmp, label %for.body, label %for.cond.cleanup, !prof !13
+
+for.body4:                                        ; preds = %for.body, %for.body4
+  %i.017 = phi i32 [ 0, %for.body ], [ %inc, %for.body4 ]
+  %idxprom = zext i32 %i.017 to i64
+  %arrayidx = getelementptr inbounds [5 x i32], [5 x i32]* @b, i64 0, i64 %idxprom
+  %0 = load i32, i32* %arrayidx, align 4, !tbaa !2
+  %mul = mul nsw i32 %0, %i.017
+  %arrayidx6 = getelementptr inbounds [5 x i32], [5 x i32]* @a, i64 0, i64 %idxprom
+  %1 = load i32, i32* %arrayidx6, align 4, !tbaa !2
+  %add = add nsw i32 %1, %mul
+  store i32 %add, i32* %arrayidx6, align 4, !tbaa !2
+  %inc = add nuw nsw i32 %i.017, 1
+  %cmp2 = icmp ult i32 %inc, 5
+  br i1 %cmp2, label %for.body4, label %for.cond.cleanup3
+}
+
+; This is negative test. Check that vectorization is not performed for COLD
+; short trip count loop requiring epilog. Note that outer loop has only 20
+; iterations and there is no associated profile info.
+
+
+; Function Attrs: uwtable
+define dso_local void @_Z3fooi3(i32 %M) local_unnamed_addr #0 !prof !11 {
+; CHECK:    [[TMP2:%.*]] = load i32, i32* [[ARRAYIDX_US:%.*]]
+; CHECK:    [[MUL_US:%.*]] = mul nsw i32 [[TMP2]], [[TMP3:%.*]]
+; CHECK:    [[TMP4:%.*]] = load i32, i32* [[ARRAYIDX6_US:%.*]]
+; CHECK:    [[ADD_US:%.*]] = add nsw i32 [[TMP4]], [[MUL_US]]
+; CHECK:    store i32 [[ADD_US]], i32* [[ARRAYIDX6_US]]
+;
+entry:
+  %a = alloca [5 x i32], align 16
+  %b = alloca [5 x i32], align 16
+  %0 = bitcast [5 x i32]* %a to i8*
+  call void @llvm.lifetime.start.p0i8(i64 20, i8* nonnull %0) #3
+  %1 = bitcast [5 x i32]* %b to i8*
+  call void @llvm.lifetime.start.p0i8(i64 20, i8* nonnull %1) #3
+  %arraydecay = getelementptr inbounds [5 x i32], [5 x i32]* %a, i64 0, i64 0
+  br label %for.body.us.preheader
+
+for.body.us.preheader:                            ; preds = %entry
+  %wide.trip.count = zext i32 %M to i64
+  br label %for.body.us
+
+for.body.us:                                      ; preds = %for.cond1.for.cond.cleanup3_crit_edge.us, %for.body.us.preheader
+  %j.019.us = phi i32 [ %inc8.us, %for.cond1.for.cond.cleanup3_crit_edge.us ], [ 0, %for.body.us.preheader ]
+  call void @_Z3barPi(i32* nonnull %arraydecay)
+  br label %for.body4.us
+
+for.body4.us:                                     ; preds = %for.body4.us, %for.body.us
+  %indvars.iv = phi i64 [ 0, %for.body.us ], [ %indvars.iv.next, %for.body4.us ]
+  %arrayidx.us = getelementptr inbounds [5 x i32], [5 x i32]* %b, i64 0, i64 %indvars.iv
+  %2 = load i32, i32* %arrayidx.us, align 4, !tbaa !2
+  %3 = trunc i64 %indvars.iv to i32
+  %mul.us = mul nsw i32 %2, %3
+  %arrayidx6.us = getelementptr inbounds [5 x i32], [5 x i32]* %a, i64 0, i64 %indvars.iv
+  %4 = load i32, i32* %arrayidx6.us, align 4, !tbaa !2
+  %add.us = add nsw i32 %4, %mul.us
+  store i32 %add.us, i32* %arrayidx6.us, align 4, !tbaa !2
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+  %exitcond = icmp eq i64 %indvars.iv.next, %wide.trip.count
+  br i1 %exitcond, label %for.cond1.for.cond.cleanup3_crit_edge.us, label %for.body4.us, !prof !14
+
+for.cond1.for.cond.cleanup3_crit_edge.us:         ; preds = %for.body4.us
+  %inc8.us = add nuw nsw i32 %j.019.us, 1
+  %exitcond21 = icmp eq i32 %inc8.us, 20
+  br i1 %exitcond21, label %for.cond.cleanup.loopexit, label %for.body.us
+
+for.cond.cleanup.loopexit:                        ; preds = %for.cond1.for.cond.cleanup3_crit_edge.us
+  br label %for.cond.cleanup
+
+for.cond.cleanup.loopexit24:                      ; preds = %for.body
+  br label %for.cond.cleanup
+
+for.cond.cleanup:                                 ; preds = %for.cond.cleanup.loopexit24, %for.cond.cleanup.loopexit
+  call void @llvm.lifetime.end.p0i8(i64 20, i8* nonnull %1) #3
+  call void @llvm.lifetime.end.p0i8(i64 20, i8* nonnull %0) #3
+  ret void
+}
+
+; Function Attrs: argmemonly nounwind willreturn
+declare void @llvm.lifetime.start.p0i8(i64 immarg, i8* nocapture) #1
+
+declare dso_local void @_Z3barPi(i32*) local_unnamed_addr
+
+; Function Attrs: argmemonly nounwind willreturn
+declare void @llvm.lifetime.end.p0i8(i64 immarg, i8* nocapture) #1
+
+attributes #0 = { "use-soft-float"="false" }
+attributes #1 = { argmemonly nounwind willreturn }
+
+!llvm.module.flags = !{!0}
+!llvm.ident = !{!1}
+
+!0 = !{i32 1, !"wchar_size", i32 4}
+!1 = !{!"clang version 10.0.0 (https://github.com/llvm/llvm-project f379dd57b978c4e1483d721f422c79e3c0c5ccdc)"}
+!2 = !{!3, !3, i64 0}
+!3 = !{!"int", !4, i64 0}
+!4 = !{!"omnipotent char", !5, i64 0}
+!5 = !{!"Simple C++ TBAA"}
+!6 = distinct !{!6, !7}
+!7 = !{!"llvm.loop.isvectorized", i32 1}
+!8 = distinct !{!8, !9, !7}
+!9 = !{!"llvm.loop.unroll.runtime.disable"}
+!10 = !{!"branch_weights", i32 999, i32 4995}
+!11 = !{!"function_entry_count", i64 1}
+!12 = !{!"branch_weights", i32 1, i32 999}
+!13 = !{!"branch_weights", i32 1000, i32 1}
+!14 = !{!"branch_weights", i32 9, i32 45}

llvm/unittests/Target/X86/MachineSizeOptsTest.cpp

TEST_F(MachineSizeOptsTest, Test) {
   auto iter = BB0.succ_begin();
   MachineBasicBlock *BB1 = *iter;
   iter++;
   MachineBasicBlock *BB2 = *iter;
   iter++;
   ASSERT_TRUE(iter == BB0.succ_end());
   MachineBasicBlock *BB3 = *BB1->succ_begin();
   ASSERT_TRUE(BB3 == *BB2->succ_begin());
-  EXPECT_FALSE(shouldOptimizeForSize(F, &PSI, MBFI_F));
-  EXPECT_TRUE(shouldOptimizeForSize(G, &PSI, MBFI_G));
-  EXPECT_FALSE(shouldOptimizeForSize(H, &PSI, MBFI_H));
-  EXPECT_FALSE(shouldOptimizeForSize(&BB0, &PSI, MBFI_F));
-  EXPECT_FALSE(shouldOptimizeForSize(BB1, &PSI, MBFI_F));
-  EXPECT_TRUE(shouldOptimizeForSize(BB2, &PSI, MBFI_F));
-  EXPECT_FALSE(shouldOptimizeForSize(BB3, &PSI, MBFI_F));
+  EXPECT_FALSE(shouldOptimizeForSize(F, &PSI, MBFI_F).getValueOr(false));
+  EXPECT_TRUE(shouldOptimizeForSize(G, &PSI, MBFI_G).getValueOr(false));
+  EXPECT_FALSE(shouldOptimizeForSize(H, &PSI, MBFI_H).getValueOr(false));
+  EXPECT_FALSE(shouldOptimizeForSize(&BB0, &PSI, MBFI_F).getValueOr(false));
+  EXPECT_FALSE(shouldOptimizeForSize(BB1, &PSI, MBFI_F).getValueOr(false));
+  EXPECT_TRUE(shouldOptimizeForSize(BB2, &PSI, MBFI_F).getValueOr(false));
+  EXPECT_FALSE(shouldOptimizeForSize(BB3, &PSI, MBFI_F).getValueOr(false));
 }
 
 const char* MachineSizeOptsTest::MIRString = R"MIR(
 --- |
   define i32 @g(i32 %x) !prof !14 {
     ret i32 0
   }
 

llvm/unittests/Transforms/Utils/SizeOptsTest.cpp

TEST_F(SizeOptsTest, Test) {
   BlockFrequencyInfo *BFI_G = BFID_G.get();
   BlockFrequencyInfo *BFI_H = BFID_H.get();
   BasicBlock &BB0 = F->getEntryBlock();
   BasicBlock *BB1 = BB0.getTerminator()->getSuccessor(0);
   BasicBlock *BB2 = BB0.getTerminator()->getSuccessor(1);
   BasicBlock *BB3 = BB1->getSingleSuccessor();
 
   EXPECT_TRUE(PSI.hasProfileSummary());
-  EXPECT_FALSE(shouldOptimizeForSize(F, &PSI, BFI_F));
-  EXPECT_TRUE(shouldOptimizeForSize(G, &PSI, BFI_G));
-  EXPECT_FALSE(shouldOptimizeForSize(H, &PSI, BFI_H));
-  EXPECT_FALSE(shouldOptimizeForSize(&BB0, &PSI, BFI_F));
-  EXPECT_FALSE(shouldOptimizeForSize(BB1, &PSI, BFI_F));
-  EXPECT_TRUE(shouldOptimizeForSize(BB2, &PSI, BFI_F));
-  EXPECT_FALSE(shouldOptimizeForSize(BB3, &PSI, BFI_F));
+  EXPECT_FALSE(shouldOptimizeForSize(F, &PSI, BFI_F).getValueOr(false));
+  EXPECT_TRUE(shouldOptimizeForSize(G, &PSI, BFI_G).getValueOr(false));
+  EXPECT_FALSE(shouldOptimizeForSize(H, &PSI, BFI_H).getValueOr(false));
+  EXPECT_FALSE(shouldOptimizeForSize(&BB0, &PSI, BFI_F).getValueOr(false));
+  EXPECT_FALSE(shouldOptimizeForSize(BB1, &PSI, BFI_F).getValueOr(false));
+  EXPECT_TRUE(shouldOptimizeForSize(BB2, &PSI, BFI_F).getValueOr(false));
+  EXPECT_FALSE(shouldOptimizeForSize(BB3, &PSI, BFI_F).getValueOr(false));
 }
 
 const char* SizeOptsTest::IRString = R"IR(
   define i32 @g(i32 %x) !prof !14 {
     ret i32 0
   }
 
   define i32 @h(i32 %x) !prof !15 {