Resubmit: [Analysis] Refactor MBB hotness/coldness into templated PSI functions

[shenhan]

Summary

The origin patch D152758 caused gcc-12 build failures, which is caused by a gcc bug, and won't be fixed in version 12.

The workaround here is to move full specialization function
definitions out of class declaration into namespace level.

Tested with linux gcc-12, linux clang, windows cl compilers.

Currently, to use PSI->isFunctionHotInCallGraph, we first need to
calculate BPI->BFI, which is expensive. Instead, we can implement
this directly with MBFI. Also as @wenlei mentioned in another patch
review, that MachineSizeOpts already has isFunctionColdInCallGraph,
isFunctionHotInCallGraphNthPercentile, etc implemented. These can be
refactored and so they can be reused across MachineFunctionSplitting
and MachineSizeOpts passes.

This CL does this - it refactors out those internal static functions
into PSI as templated functions, so they can be accessed easily.

This CL was reverted twice - 1) it hits a gcc bug; 2) it
breaks bazel's dependency rules (bazel building was not conducted
before submitting)

[wenlei]

lgtm, thanks.

[MaskRay]

Apologies but I am going to revert this patch. There is a layering violation (https://llvm.org/docs/CodingStandards.html#library-layering). LLVMAnalysis cannot depend on LLVMCodeGen. This is breaking our Bazel build.

llvm/include/llvm/Analysis/ProfileSummaryInfo.h:19:10: fatal error: 'llvm/CodeGen/MachineFunction.h' file not found
   19 | #include "llvm/CodeGen/MachineFunction.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[hokein]

llvm/include/llvm/Analysis/ProfileSummaryInfo.h

19

This seems to introduce a layering violation.

After this patch, the llvm::Analysis target now depends on the llvm::CodeGen target, but llvm::CodeGen already depends on llvm::Analysis.

One fix is to move the moving the full specialization for the MachineFunction to the CodeGen target (MachineSizeOpts.cpp)

[MaskRay]

llvm/include/llvm/Analysis/ProfileSummaryInfo.h

134

omit braces

154

omit braces

341

The LLVM convention is to omit all braces as the body has just one line.

[wenlei]

In D153927#4454956, @MaskRay wrote:

Apologies but I am going to revert this patch. There is a layering violation (https://llvm.org/docs/CodingStandards.html#library-layering). LLVMAnalysis cannot depend on LLVMCodeGen. This is breaking our Bazel build.

llvm/include/llvm/Analysis/ProfileSummaryInfo.h:19:10: fatal error: 'llvm/CodeGen/MachineFunction.h' file not found
   19 | #include "llvm/CodeGen/MachineFunction.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Good catch..

[shenhan]

The last revision Diff 8 caused bazel build failures, because ProfileSummaryInfo.h includes MachineFunction.h, which creates a cyclic dependency in bazel BUILD rules.

Fixed the issue by moving the template specialization implementation into MachineFunction.cpp, leaving only the declaration in ProfileSummaryInfo.h file.

Reran the following builds:
gcc linux
clang linux
cl windows
bazel-5.0.0 build --config=generic_clang/gcc @llvm-projectclang/...
bazel-5.0.0 build --config=generic_clang/gcc @llvm-projectclang-tools-extra/...
bazel-5.0.0 build --config=generic_clang/gcc @llvm-projectmlir/...
bazel-5.0.0 build --config=generic_clang/gcc @llvm-projectlld/...
bazel-5.0.0 build --config=generic_clang/gcc @llvm-project//llvm:CodeGen

Added @MaskRay as reviewers.

llvm/include/llvm/Analysis/ProfileSummaryInfo.h

341

Removed all cascading braces.

[shenhan]

llvm/include/llvm/Analysis/ProfileSummaryInfo.h

19

Removed the inclusion. Thanks.

[MaskRay]

Thanks. This change should support Bazel builds.

The origin CL D152758 caused gcc-12 build failures, which is caused by a gcc bug, and won't be fixed in version 12.

"CL" means https://opensource.google/documentation/reference/glossary#changelist , but our term is Differential, or just "patch" :)

In fact, I'd suggest that you remove

Below is origin CL description.

[NFC] Refactor MBB hotness/coldness into templated PSI functions.

Just start with the original description. You can add information in last paragraph about what's changed from the reverted (two?) commits.

In D152399, we calculate BPI->BFI in MachineFunctionSplit pass just to use PSI->isFunctionHotInCallGraph, which is expensive. Instead, we can implement this directly with MBFI.

Since this patch will land before D152399 , describing the situation with D152399 as the subject may be confusing. I suggest that you describe the current code structure, pretending that D152399 does not exist.
Then, if you think appropriate, describe that this refactoring will make D152399 more focused.

llvm/lib/CodeGen/MachineFunction.cpp

1504

Remove namespace llvm (see https://llvm.org/docs/CodingStandards.html#use-namespace-qualifiers-to-implement-previously-declared-functions).

ProfileSummaryInfo is in the namespace llvm, so ProfileSummaryInfo::getEntryCount is sufficient.

Ideally getEntryCount should be a const reference to indicate that it cannot be null, but the API uses a pointer, so it seems fine to not change it.

[shenhan]

In D153927#4458106, @MaskRay wrote:

Thanks. This change should support Bazel builds.

The origin CL D152758 caused gcc-12 build failures, which is caused by a gcc bug, and won't be fixed in version 12.

"CL" means https://opensource.google/documentation/reference/glossary#changelist , but our term is Differential, or just "patch" :)

Done.

In fact, I'd suggest that you remove

Below is origin CL description.

[NFC] Refactor MBB hotness/coldness into templated PSI functions.

Just start with the original description. You can add information in last paragraph about what's changed from the reverted (two?) commits.

Done

In D152399, we calculate BPI->BFI in MachineFunctionSplit pass just to use PSI->isFunctionHotInCallGraph, which is expensive. Instead, we can implement this directly with MBFI.

Since this patch will land before D152399 , describing the situation with D152399 as the subject may be confusing. I suggest that you describe the current code structure, pretending that D152399 does not exist.
Then, if you think appropriate, describe that this refactoring will make D152399 more focused.

Revised the description.

llvm/include/llvm/Analysis/ProfileSummaryInfo.h

 // information.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_ANALYSIS_PROFILESUMMARYINFO_H
 #define LLVM_ANALYSIS_PROFILESUMMARYINFO_H
 
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/Analysis/BlockFrequencyInfo.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/Instructions.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/IR/ProfileSummary.h"
 #include "llvm/Pass.h"
 #include <memory>
 #include <optional>
 
 namespace llvm {
 class BasicBlock;
-class BlockFrequencyInfo;
 class CallBase;
-class Function;
+class MachineFunction;
 
 /// Analysis providing profile information.
 ///
 /// This is an immutable analysis pass that provides ability to query global
 /// (program-level) profile information. The main APIs are isHotCount and
 /// isColdCount that tells whether a given profile count is considered hot/cold
 /// based on the profile summary. This also provides convenience methods to
 /// check whether a function is hot or cold.
std::optional<uint64_t> getProfileCount(const CallBase &CallInst,
                                           BlockFrequencyInfo *BFI,
                                           bool AllowSynthetic = false) const;
   /// Returns true if module \c M has partial-profile sample profile.
   bool hasPartialSampleProfile() const;
   /// Returns true if the working set size of the code is considered huge.
   bool hasHugeWorkingSetSize() const;
   /// Returns true if the working set size of the code is considered large.
   bool hasLargeWorkingSetSize() const;
-  /// Returns true if \p F has hot function entry.
-  bool isFunctionEntryHot(const Function *F) const;
+  /// Returns true if \p F has hot function entry. If it returns false, it
+  /// either means it is not hot or it is unknown whether it is hot or not (for
+  /// example, no profile data is available).
+  template <typename FuncT> bool isFunctionEntryHot(const FuncT *F) const {
+    if (!F || !hasProfileSummary())
+      return false;
+    std::optional<Function::ProfileCount> FunctionCount = getEntryCount(F);
+    // FIXME: The heuristic used below for determining hotness is based on
+    // preliminary SPEC tuning for inliner. This will eventually be a
+    // convenience method that calls isHotCount.
+    return FunctionCount && isHotCount(FunctionCount->getCount());
+  }
+
   /// Returns true if \p F contains hot code.
-  bool isFunctionHotInCallGraph(const Function *F,
-                                BlockFrequencyInfo &BFI) const;
+  template <typename FuncT, typename BFIT>
+  bool isFunctionHotInCallGraph(const FuncT *F, BFIT &BFI) const {
+    if (!F || !hasProfileSummary())
+      return false;
+    if (auto FunctionCount = getEntryCount(F))
+      if (isHotCount(FunctionCount->getCount()))
+        return true;
+
+    if (auto TotalCallCount = getTotalCallCount(F))
+      if (isHotCount(*TotalCallCount))
+        return true;
+
+    for (const auto &BB : *F)
+      if (isHotBlock(&BB, &BFI))
+        return true;
+    return false;
+  }
   /// Returns true if \p F has cold function entry.
   bool isFunctionEntryCold(const Function *F) const;
   /// Returns true if \p F contains only cold code.
-  bool isFunctionColdInCallGraph(const Function *F,
-                                 BlockFrequencyInfo &BFI) const;
+  template <typename FuncT, typename BFIT>
+  bool isFunctionColdInCallGraph(const FuncT *F, BFIT &BFI) const {
+    if (!F || !hasProfileSummary())
+      return false;
+    if (auto FunctionCount = getEntryCount(F))
+      if (!isColdCount(FunctionCount->getCount()))
+        return false;
+
+    if (auto TotalCallCount = getTotalCallCount(F))
+      if (!isColdCount(*TotalCallCount))
+        return false;
+
+    for (const auto &BB : *F)
+      if (!isColdBlock(&BB, &BFI))
+        return false;
+    return true;
+  }
   /// Returns true if the hotness of \p F is unknown.
   bool isFunctionHotnessUnknown(const Function &F) const;
   /// Returns true if \p F contains hot code with regard to a given hot
   /// percentile cutoff value.
+  template <typename FuncT, typename BFIT>
   bool isFunctionHotInCallGraphNthPercentile(int PercentileCutoff,
-                                             const Function *F,
-                                             BlockFrequencyInfo &BFI) const;
+                                             const FuncT *F, BFIT &BFI) const {
+    return isFunctionHotOrColdInCallGraphNthPercentile<true, FuncT, BFIT>(
+        PercentileCutoff, F, BFI);
+  }
   /// Returns true if \p F contains cold code with regard to a given cold
   /// percentile cutoff value.
+  template <typename FuncT, typename BFIT>
   bool isFunctionColdInCallGraphNthPercentile(int PercentileCutoff,
-                                              const Function *F,
-                                              BlockFrequencyInfo &BFI) const;
+                                              const FuncT *F, BFIT &BFI) const {
+    return isFunctionHotOrColdInCallGraphNthPercentile<false, FuncT, BFIT>(
+        PercentileCutoff, F, BFI);
+  }
   /// Returns true if count \p C is considered hot.
   bool isHotCount(uint64_t C) const;
   /// Returns true if count \p C is considered cold.
   bool isColdCount(uint64_t C) const;
   /// Returns true if count \p C is considered hot with regard to a given
   /// hot percentile cutoff value.
   /// PercentileCutoff is encoded as a 6 digit decimal fixed point number, where
   /// the first two digits are the whole part. E.g. 995000 for 99.5 percentile.
   bool isHotCountNthPercentile(int PercentileCutoff, uint64_t C) const;
   /// Returns true if count \p C is considered cold with regard to a given
   /// cold percentile cutoff value.
   /// PercentileCutoff is encoded as a 6 digit decimal fixed point number, where
   /// the first two digits are the whole part. E.g. 995000 for 99.5 percentile.
   bool isColdCountNthPercentile(int PercentileCutoff, uint64_t C) const;
+
   /// Returns true if BasicBlock \p BB is considered hot.
-  bool isHotBlock(const BasicBlock *BB, BlockFrequencyInfo *BFI) const;
+  template <typename BBType, typename BFIT>
+  bool isHotBlock(const BBType *BB, BFIT *BFI) const {
+    auto Count = BFI->getBlockProfileCount(BB);
+    return Count && isHotCount(*Count);
+  }
+
   /// Returns true if BasicBlock \p BB is considered cold.
-  bool isColdBlock(const BasicBlock *BB, BlockFrequencyInfo *BFI) const;
-  /// Returns true if BasicBlock \p BB is considered hot with regard to a given
-  /// hot percentile cutoff value.
-  /// PercentileCutoff is encoded as a 6 digit decimal fixed point number, where
-  /// the first two digits are the whole part. E.g. 995000 for 99.5 percentile.
-  bool isHotBlockNthPercentile(int PercentileCutoff, const BasicBlock *BB,
-                               BlockFrequencyInfo *BFI) const;
+  template <typename BBType, typename BFIT>
+  bool isColdBlock(const BBType *BB, BFIT *BFI) const {
+    auto Count = BFI->getBlockProfileCount(BB);
+    return Count && isColdCount(*Count);
+  }
+
+  template <typename BFIT>
+  bool isColdBlock(BlockFrequency BlockFreq, const BFIT *BFI) const {
+    auto Count = BFI->getProfileCountFromFreq(BlockFreq.getFrequency());
+    return Count && isColdCount(*Count);
+  }
+
+  template <typename BBType, typename BFIT>
+  bool isHotBlockNthPercentile(int PercentileCutoff, const BBType *BB,
+                               BFIT *BFI) const {
+    return isHotOrColdBlockNthPercentile<true, BBType, BFIT>(PercentileCutoff,
+                                                             BB, BFI);
+  }
+
+  template <typename BFIT>
+  bool isHotBlockNthPercentile(int PercentileCutoff, BlockFrequency BlockFreq,
+                               BFIT *BFI) const {
+    return isHotOrColdBlockNthPercentile<true, BFIT>(PercentileCutoff,
+                                                     BlockFreq, BFI);
+  }
+
   /// Returns true if BasicBlock \p BB is considered cold with regard to a given
   /// cold percentile cutoff value.
   /// PercentileCutoff is encoded as a 6 digit decimal fixed point number, where
   /// the first two digits are the whole part. E.g. 995000 for 99.5 percentile.
-  bool isColdBlockNthPercentile(int PercentileCutoff, const BasicBlock *BB,
-                                BlockFrequencyInfo *BFI) const;
+  template <typename BBType, typename BFIT>
+  bool isColdBlockNthPercentile(int PercentileCutoff, const BBType *BB,
+                                BFIT *BFI) const {
+    return isHotOrColdBlockNthPercentile<false, BBType, BFIT>(PercentileCutoff,
+                                                              BB, BFI);
+  }
+  template <typename BFIT>
+  bool isColdBlockNthPercentile(int PercentileCutoff, BlockFrequency BlockFreq,
+                                BFIT *BFI) const {
+    return isHotOrColdBlockNthPercentile<false, BFIT>(PercentileCutoff,
+                                                      BlockFreq, BFI);
+  }
   /// Returns true if the call site \p CB is considered hot.
   bool isHotCallSite(const CallBase &CB, BlockFrequencyInfo *BFI) const;
   /// Returns true if call site \p CB is considered cold.
   bool isColdCallSite(const CallBase &CB, BlockFrequencyInfo *BFI) const;
   /// Returns HotCountThreshold if set. Recompute HotCountThreshold
   /// if not set.
   uint64_t getOrCompHotCountThreshold() const;
   /// Returns ColdCountThreshold if set. Recompute HotCountThreshold
   /// if not set.
   uint64_t getOrCompColdCountThreshold() const;
   /// Returns HotCountThreshold if set.
   uint64_t getHotCountThreshold() const {
     return HotCountThreshold.value_or(0);
   }
   /// Returns ColdCountThreshold if set.
   uint64_t getColdCountThreshold() const {
     return ColdCountThreshold.value_or(0);
   }
 
- private:
-   template <bool isHot>
-   bool isFunctionHotOrColdInCallGraphNthPercentile(
-       int PercentileCutoff, const Function *F, BlockFrequencyInfo &BFI) const;
-   template <bool isHot>
-   bool isHotOrColdCountNthPercentile(int PercentileCutoff, uint64_t C) const;
-   template <bool isHot>
-   bool isHotOrColdBlockNthPercentile(int PercentileCutoff,
-                                      const BasicBlock *BB,
-                                      BlockFrequencyInfo *BFI) const;
+  template <typename FuncT>
+  std::optional<uint64_t> getTotalCallCount(const FuncT *F) const {
+    return std::nullopt;
+  }
+
+  template <bool isHot, typename FuncT, typename BFIT>
+  bool isFunctionHotOrColdInCallGraphNthPercentile(int PercentileCutoff,
+                                                   const FuncT *F,
+                                                   BFIT &FI) const {
+    if (!F || !hasProfileSummary())
+      return false;
+    if (auto FunctionCount = getEntryCount(F)) {
+      if (isHot &&
+          isHotCountNthPercentile(PercentileCutoff, FunctionCount->getCount()))
+        return true;
+      if (!isHot && !isColdCountNthPercentile(PercentileCutoff,
+                                              FunctionCount->getCount()))
+        return false;
+    }
+    if (auto TotalCallCount = getTotalCallCount(F)) {
+      if (isHot && isHotCountNthPercentile(PercentileCutoff, *TotalCallCount))
+        return true;
+      if (!isHot &&
+          !isColdCountNthPercentile(PercentileCutoff, *TotalCallCount))
+        return false;
+    }
+    for (const auto &BB : *F) {
+      if (isHot && isHotBlockNthPercentile(PercentileCutoff, &BB, &FI))
+        return true;
+      if (!isHot && !isColdBlockNthPercentile(PercentileCutoff, &BB, &FI))
+        return false;
+    }
+    return !isHot;
+  }
+
+
+  template <bool isHot, typename BBType, typename BFIT>
+  bool isHotOrColdBlockNthPercentile(int PercentileCutoff, const BBType *BB,
+                                     BFIT *BFI) const {
+    auto Count = BFI->getBlockProfileCount(BB);
+    if (isHot)
+      return Count && isHotCountNthPercentile(PercentileCutoff, *Count);
+    else
+      return Count && isColdCountNthPercentile(PercentileCutoff, *Count);
+  }
+
+  template <bool isHot, typename BFIT>
+                                     BlockFrequency BlockFreq,
+                                     BFIT *BFI) const {
+    auto Count = BFI->getProfileCountFromFreq(BlockFreq.getFrequency());
+    if (isHot)
+      return Count && isHotCountNthPercentile(PercentileCutoff, *Count);
+    else
+      return Count && isColdCountNthPercentile(PercentileCutoff, *Count);
+  }
+
+  template <typename FuncT>
+  std::optional<Function::ProfileCount> getEntryCount(const FuncT *F) const {
+    return F->getEntryCount();
+  }
 };
 
+template <>
+inline std::optional<uint64_t>
+ProfileSummaryInfo::getTotalCallCount<Function>(const Function *F) const {
+  if (!hasSampleProfile())
+    return std::nullopt;
+  uint64_t TotalCallCount = 0;
+  for (const auto &BB : *F)
+    for (const auto &I : BB)
+      if (isa<CallInst>(I) || isa<InvokeInst>(I))
+        if (auto CallCount = getProfileCount(cast<CallBase>(I), nullptr))
+          TotalCallCount += *CallCount;
+  return TotalCallCount;
+}
+
+// Declare template specialization for llvm::MachineFunction. Do not implement
+// here, because we cannot include MachineFunction header here, that would break
+// dependency rules.
+template <>
+std::optional<Function::ProfileCount>
+ProfileSummaryInfo::getEntryCount<MachineFunction>(
+    const MachineFunction *F) const;
+
 /// An analysis pass based on legacy pass manager to deliver ProfileSummaryInfo.
 class ProfileSummaryInfoWrapperPass : public ImmutablePass {
   std::unique_ptr<ProfileSummaryInfo> PSI;
 
 public:
   static char ID;
   ProfileSummaryInfoWrapperPass();
 

llvm/include/llvm/Transforms/Utils/SizeOpts.h

return PGSOColdCodeOnly ||
          (PSI->hasInstrumentationProfile() && PGSOColdCodeOnlyForInstrPGO) ||
          (PSI->hasSampleProfile() &&
           ((!PSI->hasPartialSampleProfile() && PGSOColdCodeOnlyForSamplePGO) ||
            (PSI->hasPartialSampleProfile() &&
             PGSOColdCodeOnlyForPartialSamplePGO))) ||
          (PGSOLargeWorkingSetSizeOnly && !PSI->hasLargeWorkingSetSize());
 }
 
-template<typename AdapterT, typename FuncT, typename BFIT>
+template <typename FuncT, typename BFIT>
 bool shouldFuncOptimizeForSizeImpl(const FuncT *F, ProfileSummaryInfo *PSI,
                                    BFIT *BFI, PGSOQueryType QueryType) {
   assert(F);
   if (!PSI || !BFI || !PSI->hasProfileSummary())
     return false;
   if (ForcePGSO)
     return true;
   if (!EnablePGSO)
     return false;
   if (isPGSOColdCodeOnly(PSI))
-    return AdapterT::isFunctionColdInCallGraph(F, PSI, *BFI);
+    return PSI->isFunctionColdInCallGraph(F, *BFI);
   if (PSI->hasSampleProfile())
     // The "isCold" check seems to work better for Sample PGO as it could have
     // many profile-unannotated functions.
-    return AdapterT::isFunctionColdInCallGraphNthPercentile(
-        PgsoCutoffSampleProf, F, PSI, *BFI);
-  return !AdapterT::isFunctionHotInCallGraphNthPercentile(PgsoCutoffInstrProf,
-                                                          F, PSI, *BFI);
+    return PSI->isFunctionColdInCallGraphNthPercentile(PgsoCutoffSampleProf, F,
+                                                       *BFI);
+  return !PSI->isFunctionHotInCallGraphNthPercentile(PgsoCutoffInstrProf, F,
+                                                     *BFI);
 }
 
-template<typename AdapterT, typename BlockTOrBlockFreq, typename BFIT>
-bool shouldOptimizeForSizeImpl(BlockTOrBlockFreq BBOrBlockFreq, ProfileSummaryInfo *PSI,
-                               BFIT *BFI, PGSOQueryType QueryType) {
+template <typename BlockTOrBlockFreq, typename BFIT>
+bool shouldOptimizeForSizeImpl(BlockTOrBlockFreq BBOrBlockFreq,
+                               ProfileSummaryInfo *PSI, BFIT *BFI,
+                               PGSOQueryType QueryType) {
   if (!PSI || !BFI || !PSI->hasProfileSummary())
     return false;
   if (ForcePGSO)
     return true;
   if (!EnablePGSO)
     return false;
   if (isPGSOColdCodeOnly(PSI))
-    return AdapterT::isColdBlock(BBOrBlockFreq, PSI, BFI);
+    return PSI->isColdBlock(BBOrBlockFreq, BFI);
   if (PSI->hasSampleProfile())
     // The "isCold" check seems to work better for Sample PGO as it could have
     // many profile-unannotated functions.
-    return AdapterT::isColdBlockNthPercentile(PgsoCutoffSampleProf,
-                                              BBOrBlockFreq, PSI, BFI);
-  return !AdapterT::isHotBlockNthPercentile(PgsoCutoffInstrProf, BBOrBlockFreq,
-                                            PSI, BFI);
+    return PSI->isColdBlockNthPercentile(PgsoCutoffSampleProf, BBOrBlockFreq,
+                                         BFI);
+  return !PSI->isHotBlockNthPercentile(PgsoCutoffInstrProf, BBOrBlockFreq, BFI);
 }
 
 /// Returns true if function \p F is suggested to be size-optimized based on the
 /// profile.
 bool shouldOptimizeForSize(const Function *F, ProfileSummaryInfo *PSI,
                            BlockFrequencyInfo *BFI,
                            PGSOQueryType QueryType = PGSOQueryType::Other);
 

llvm/lib/Analysis/ProfileSummaryInfo.cpp

if (Call.extractProfTotalWeight(TotalCount))
       return TotalCount;
     return std::nullopt;
   }
   if (BFI)
     return BFI->getBlockProfileCount(Call.getParent(), AllowSynthetic);
   return std::nullopt;
 }
 
-/// Returns true if the function's entry is hot. If it returns false, it
-/// either means it is not hot or it is unknown whether it is hot or not (for
-/// example, no profile data is available).
-bool ProfileSummaryInfo::isFunctionEntryHot(const Function *F) const {
-  if (!F || !hasProfileSummary())
-    return false;
-  auto FunctionCount = F->getEntryCount();
-  // FIXME: The heuristic used below for determining hotness is based on
-  // preliminary SPEC tuning for inliner. This will eventually be a
-  // convenience method that calls isHotCount.
-  return FunctionCount && isHotCount(FunctionCount->getCount());
-}
-
-/// Returns true if the function contains hot code. This can include a hot
-/// function entry count, hot basic block, or (in the case of Sample PGO)
-/// hot total call edge count.
-/// If it returns false, it either means it is not hot or it is unknown
-/// (for example, no profile data is available).
-bool ProfileSummaryInfo::isFunctionHotInCallGraph(
-    const Function *F, BlockFrequencyInfo &BFI) const {
-  if (!F || !hasProfileSummary())
-    return false;
-  if (auto FunctionCount = F->getEntryCount())
-    if (isHotCount(FunctionCount->getCount()))
-      return true;
-
-  if (hasSampleProfile()) {
-    uint64_t TotalCallCount = 0;
-    for (const auto &BB : *F)
-      for (const auto &I : BB)
-        if (isa<CallInst>(I) || isa<InvokeInst>(I))
-          if (auto CallCount = getProfileCount(cast<CallBase>(I), nullptr))
-            TotalCallCount += *CallCount;
-    if (isHotCount(TotalCallCount))
-      return true;
-  }
-  for (const auto &BB : *F)
-    if (isHotBlock(&BB, &BFI))
-      return true;
-  return false;
-}
-
-/// Returns true if the function only contains cold code. This means that
-/// the function entry and blocks are all cold, and (in the case of Sample PGO)
-/// the total call edge count is cold.
-/// If it returns false, it either means it is not cold or it is unknown
-/// (for example, no profile data is available).
-bool ProfileSummaryInfo::isFunctionColdInCallGraph(
-    const Function *F, BlockFrequencyInfo &BFI) const {
-  if (!F || !hasProfileSummary())
-    return false;
-  if (auto FunctionCount = F->getEntryCount())
-    if (!isColdCount(FunctionCount->getCount()))
-      return false;
-
-  if (hasSampleProfile()) {
-    uint64_t TotalCallCount = 0;
-    for (const auto &BB : *F)
-      for (const auto &I : BB)
-        if (isa<CallInst>(I) || isa<InvokeInst>(I))
-          if (auto CallCount = getProfileCount(cast<CallBase>(I), nullptr))
-            TotalCallCount += *CallCount;
-    if (!isColdCount(TotalCallCount))
-      return false;
-  }
-  for (const auto &BB : *F)
-    if (!isColdBlock(&BB, &BFI))
-      return false;
-  return true;
-}
-
 bool ProfileSummaryInfo::isFunctionHotnessUnknown(const Function &F) const {
   assert(hasPartialSampleProfile() && "Expect partial sample profile");
   return !F.getEntryCount();
 }
 
-template <bool isHot>
-bool ProfileSummaryInfo::isFunctionHotOrColdInCallGraphNthPercentile(
-    int PercentileCutoff, const Function *F, BlockFrequencyInfo &BFI) const {
-  if (!F || !hasProfileSummary())
-    return false;
-  if (auto FunctionCount = F->getEntryCount()) {
-    if (isHot &&
-        isHotCountNthPercentile(PercentileCutoff, FunctionCount->getCount()))
-      return true;
-    if (!isHot &&
-        !isColdCountNthPercentile(PercentileCutoff, FunctionCount->getCount()))
-      return false;
-  }
-  if (hasSampleProfile()) {
-    uint64_t TotalCallCount = 0;
-    for (const auto &BB : *F)
-      for (const auto &I : BB)
-        if (isa<CallInst>(I) || isa<InvokeInst>(I))
-          if (auto CallCount = getProfileCount(cast<CallBase>(I), nullptr))
-            TotalCallCount += *CallCount;
-    if (isHot && isHotCountNthPercentile(PercentileCutoff, TotalCallCount))
-      return true;
-    if (!isHot && !isColdCountNthPercentile(PercentileCutoff, TotalCallCount))
-      return false;
-  }
-  for (const auto &BB : *F) {
-    if (isHot && isHotBlockNthPercentile(PercentileCutoff, &BB, &BFI))
-      return true;
-    if (!isHot && !isColdBlockNthPercentile(PercentileCutoff, &BB, &BFI))
-      return false;
-  }
-  return !isHot;
-}
-
-// Like isFunctionHotInCallGraph but for a given cutoff.
-bool ProfileSummaryInfo::isFunctionHotInCallGraphNthPercentile(
-    int PercentileCutoff, const Function *F, BlockFrequencyInfo &BFI) const {
-  return isFunctionHotOrColdInCallGraphNthPercentile<true>(
-      PercentileCutoff, F, BFI);
-}
-
-bool ProfileSummaryInfo::isFunctionColdInCallGraphNthPercentile(
-    int PercentileCutoff, const Function *F, BlockFrequencyInfo &BFI) const {
-  return isFunctionHotOrColdInCallGraphNthPercentile<false>(
-      PercentileCutoff, F, BFI);
-}
-
 /// Returns true if the function's entry is a cold. If it returns false, it
 /// either means it is not cold or it is unknown whether it is cold or not (for
 /// example, no profile data is available).
 bool ProfileSummaryInfo::isFunctionEntryCold(const Function *F) const {
   if (!F)
     return false;
   if (F->hasFnAttribute(Attribute::Cold))
     return true;

 uint64_t ProfileSummaryInfo::getOrCompHotCountThreshold() const {
   return HotCountThreshold.value_or(UINT64_MAX);
 }
 
 uint64_t ProfileSummaryInfo::getOrCompColdCountThreshold() const {
   return ColdCountThreshold.value_or(0);
 }
 
-bool ProfileSummaryInfo::isHotBlock(const BasicBlock *BB,
-                                    BlockFrequencyInfo *BFI) const {
-  auto Count = BFI->getBlockProfileCount(BB);
-  return Count && isHotCount(*Count);
-}
-
-bool ProfileSummaryInfo::isColdBlock(const BasicBlock *BB,
-                                     BlockFrequencyInfo *BFI) const {
-  auto Count = BFI->getBlockProfileCount(BB);
-  return Count && isColdCount(*Count);
-}
-
-template <bool isHot>
-bool ProfileSummaryInfo::isHotOrColdBlockNthPercentile(
-    int PercentileCutoff, const BasicBlock *BB, BlockFrequencyInfo *BFI) const {
-  auto Count = BFI->getBlockProfileCount(BB);
-  if (isHot)
-    return Count && isHotCountNthPercentile(PercentileCutoff, *Count);
-  else
-    return Count && isColdCountNthPercentile(PercentileCutoff, *Count);
-}
-
-bool ProfileSummaryInfo::isHotBlockNthPercentile(
-    int PercentileCutoff, const BasicBlock *BB, BlockFrequencyInfo *BFI) const {
-  return isHotOrColdBlockNthPercentile<true>(PercentileCutoff, BB, BFI);
-}
-
-bool ProfileSummaryInfo::isColdBlockNthPercentile(
-    int PercentileCutoff, const BasicBlock *BB, BlockFrequencyInfo *BFI) const {
-  return isHotOrColdBlockNthPercentile<false>(PercentileCutoff, BB, BFI);
-}
-
 bool ProfileSummaryInfo::isHotCallSite(const CallBase &CB,
                                        BlockFrequencyInfo *BFI) const {
   auto C = getProfileCount(CB, BFI);
   return C && isHotCount(*C);
 }
 
 bool ProfileSummaryInfo::isColdCallSite(const CallBase &CB,
                                         BlockFrequencyInfo *BFI) const {

llvm/lib/CodeGen/MachineFunction.cpp

 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Analysis/ConstantFolding.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineJumpTableInfo.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
if (Constants[i].isMachineConstantPoolEntry())
       Constants[i].Val.MachineCPVal->print(OS);
     else
       Constants[i].Val.ConstVal->printAsOperand(OS, /*PrintType=*/false);
     OS << ", align=" << Constants[i].getAlign().value();
     OS << "\n";
   }
 }
 
+//===----------------------------------------------------------------------===//
+// Template specialization for MachineFunction implementation of
+// ProfileSummaryInfo::getEntryCount().
+//===----------------------------------------------------------------------===//
+template <>
+std::optional<Function::ProfileCount>
+ProfileSummaryInfo::getEntryCount<llvm::MachineFunction>(
+    const llvm::MachineFunction *F) const {
+  return F->getFunction().getEntryCount();
+}
+
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 LLVM_DUMP_METHOD void MachineConstantPool::dump() const { print(dbgs()); }
 #endif

llvm/lib/CodeGen/MachineSizeOpts.cpp

 using namespace llvm;
 
 extern cl::opt<bool> EnablePGSO;
 extern cl::opt<bool> PGSOLargeWorkingSetSizeOnly;
 extern cl::opt<bool> ForcePGSO;
 extern cl::opt<int> PgsoCutoffInstrProf;
 extern cl::opt<int> PgsoCutoffSampleProf;
 
-namespace {
-namespace machine_size_opts_detail {
-
-/// Like ProfileSummaryInfo::isColdBlock but for MachineBasicBlock.
-bool isColdBlock(const MachineBasicBlock *MBB,
-                 ProfileSummaryInfo *PSI,
-                 const MachineBlockFrequencyInfo *MBFI) {
-  auto Count = MBFI->getBlockProfileCount(MBB);
-  return Count && PSI->isColdCount(*Count);
-}
-
-bool isColdBlock(BlockFrequency BlockFreq,
-                 ProfileSummaryInfo *PSI,
-                 const MachineBlockFrequencyInfo *MBFI) {
-  auto Count = MBFI->getProfileCountFromFreq(BlockFreq.getFrequency());
-  return Count && PSI->isColdCount(*Count);
-}
-
-/// Like ProfileSummaryInfo::isHotBlockNthPercentile but for MachineBasicBlock.
-static bool isHotBlockNthPercentile(int PercentileCutoff,
-                                    const MachineBasicBlock *MBB,
-                                    ProfileSummaryInfo *PSI,
-                                    const MachineBlockFrequencyInfo *MBFI) {
-  auto Count = MBFI->getBlockProfileCount(MBB);
-  return Count && PSI->isHotCountNthPercentile(PercentileCutoff, *Count);
-}
-
-static bool isHotBlockNthPercentile(int PercentileCutoff,
-                                    BlockFrequency BlockFreq,
-                                    ProfileSummaryInfo *PSI,
-                                    const MachineBlockFrequencyInfo *MBFI) {
-  auto Count = MBFI->getProfileCountFromFreq(BlockFreq.getFrequency());
-  return Count && PSI->isHotCountNthPercentile(PercentileCutoff, *Count);
-}
-
-static bool isColdBlockNthPercentile(int PercentileCutoff,
-                                     const MachineBasicBlock *MBB,
-                                     ProfileSummaryInfo *PSI,
-                                     const MachineBlockFrequencyInfo *MBFI) {
-  auto Count = MBFI->getBlockProfileCount(MBB);
-  return Count && PSI->isColdCountNthPercentile(PercentileCutoff, *Count);
-}
-
-static bool isColdBlockNthPercentile(int PercentileCutoff,
-                                     BlockFrequency BlockFreq,
-                                     ProfileSummaryInfo *PSI,
-                                     const MachineBlockFrequencyInfo *MBFI) {
-  auto Count = MBFI->getProfileCountFromFreq(BlockFreq.getFrequency());
-  return Count && PSI->isColdCountNthPercentile(PercentileCutoff, *Count);
-}
-
-/// Like ProfileSummaryInfo::isFunctionColdInCallGraph but for
-/// MachineFunction.
-bool isFunctionColdInCallGraph(
-    const MachineFunction *MF,
-    ProfileSummaryInfo *PSI,
-    const MachineBlockFrequencyInfo &MBFI) {
-  if (auto FunctionCount = MF->getFunction().getEntryCount())
-    if (!PSI->isColdCount(FunctionCount->getCount()))
-      return false;
-  for (const auto &MBB : *MF)
-    if (!isColdBlock(&MBB, PSI, &MBFI))
-      return false;
-  return true;
-}
-
-/// Like ProfileSummaryInfo::isFunctionHotInCallGraphNthPercentile but for
-/// MachineFunction.
-bool isFunctionHotInCallGraphNthPercentile(
-    int PercentileCutoff,
-    const MachineFunction *MF,
-    ProfileSummaryInfo *PSI,
-    const MachineBlockFrequencyInfo &MBFI) {
-  if (auto FunctionCount = MF->getFunction().getEntryCount())
-    if (PSI->isHotCountNthPercentile(PercentileCutoff,
-                                     FunctionCount->getCount()))
-      return true;
-  for (const auto &MBB : *MF)
-    if (isHotBlockNthPercentile(PercentileCutoff, &MBB, PSI, &MBFI))
-      return true;
-  return false;
-}
-
-bool isFunctionColdInCallGraphNthPercentile(
-    int PercentileCutoff, const MachineFunction *MF, ProfileSummaryInfo *PSI,
-    const MachineBlockFrequencyInfo &MBFI) {
-  if (auto FunctionCount = MF->getFunction().getEntryCount())
-    if (!PSI->isColdCountNthPercentile(PercentileCutoff,
-                                       FunctionCount->getCount()))
-      return false;
-  for (const auto &MBB : *MF)
-    if (!isColdBlockNthPercentile(PercentileCutoff, &MBB, PSI, &MBFI))
-      return false;
-  return true;
-}
-} // namespace machine_size_opts_detail
-
-struct MachineBasicBlockBFIAdapter {
-  static bool isFunctionColdInCallGraph(const MachineFunction *MF,
-                                        ProfileSummaryInfo *PSI,
-                                        const MachineBlockFrequencyInfo &MBFI) {
-    return machine_size_opts_detail::isFunctionColdInCallGraph(MF, PSI, MBFI);
-  }
-  static bool isFunctionHotInCallGraphNthPercentile(
-      int CutOff,
-      const MachineFunction *MF,
-      ProfileSummaryInfo *PSI,
-      const MachineBlockFrequencyInfo &MBFI) {
-    return machine_size_opts_detail::isFunctionHotInCallGraphNthPercentile(
-        CutOff, MF, PSI, MBFI);
-  }
-  static bool isFunctionColdInCallGraphNthPercentile(
-      int CutOff, const MachineFunction *MF, ProfileSummaryInfo *PSI,
-      const MachineBlockFrequencyInfo &MBFI) {
-    return machine_size_opts_detail::isFunctionColdInCallGraphNthPercentile(
-        CutOff, MF, PSI, MBFI);
-  }
-  static bool isColdBlock(const MachineBasicBlock *MBB,
-                          ProfileSummaryInfo *PSI,
-                          const MachineBlockFrequencyInfo *MBFI) {
-    return machine_size_opts_detail::isColdBlock(MBB, PSI, MBFI);
-  }
-  static bool isColdBlock(BlockFrequency BlockFreq,
-                          ProfileSummaryInfo *PSI,
-                          const MachineBlockFrequencyInfo *MBFI) {
-    return machine_size_opts_detail::isColdBlock(BlockFreq, PSI, MBFI);
-  }
-  static bool isHotBlockNthPercentile(int CutOff,
-                                      const MachineBasicBlock *MBB,
-                                      ProfileSummaryInfo *PSI,
-                                      const MachineBlockFrequencyInfo *MBFI) {
-    return machine_size_opts_detail::isHotBlockNthPercentile(
-        CutOff, MBB, PSI, MBFI);
-  }
-  static bool isHotBlockNthPercentile(int CutOff,
-                                      BlockFrequency BlockFreq,
-                                      ProfileSummaryInfo *PSI,
-                                      const MachineBlockFrequencyInfo *MBFI) {
-    return machine_size_opts_detail::isHotBlockNthPercentile(
-        CutOff, BlockFreq, PSI, MBFI);
-  }
-  static bool isColdBlockNthPercentile(int CutOff, const MachineBasicBlock *MBB,
-                                       ProfileSummaryInfo *PSI,
-                                       const MachineBlockFrequencyInfo *MBFI) {
-    return machine_size_opts_detail::isColdBlockNthPercentile(CutOff, MBB, PSI,
-                                                              MBFI);
-  }
-  static bool isColdBlockNthPercentile(int CutOff, BlockFrequency BlockFreq,
-                                       ProfileSummaryInfo *PSI,
-                                       const MachineBlockFrequencyInfo *MBFI) {
-    return machine_size_opts_detail::isColdBlockNthPercentile(CutOff, BlockFreq,
-                                                              PSI, MBFI);
-  }
-};
-} // end anonymous namespace
-
 bool llvm::shouldOptimizeForSize(const MachineFunction *MF,
                                  ProfileSummaryInfo *PSI,
                                  const MachineBlockFrequencyInfo *MBFI,
                                  PGSOQueryType QueryType) {
-  return shouldFuncOptimizeForSizeImpl<MachineBasicBlockBFIAdapter>(
-      MF, PSI, MBFI, QueryType);
+  return shouldFuncOptimizeForSizeImpl(MF, PSI, MBFI, QueryType);
 }
 
 bool llvm::shouldOptimizeForSize(const MachineBasicBlock *MBB,
                                  ProfileSummaryInfo *PSI,
                                  const MachineBlockFrequencyInfo *MBFI,
                                  PGSOQueryType QueryType) {
   assert(MBB);
-  return shouldOptimizeForSizeImpl<MachineBasicBlockBFIAdapter>(
-      MBB, PSI, MBFI, QueryType);
+  return shouldOptimizeForSizeImpl(MBB, PSI, MBFI, QueryType);
 }
 
 bool llvm::shouldOptimizeForSize(const MachineBasicBlock *MBB,
                                  ProfileSummaryInfo *PSI,
                                  MBFIWrapper *MBFIW,
                                  PGSOQueryType QueryType) {
   assert(MBB);
   if (!PSI || !MBFIW)
     return false;
   BlockFrequency BlockFreq = MBFIW->getBlockFreq(MBB);
-  return shouldOptimizeForSizeImpl<MachineBasicBlockBFIAdapter>(
-      BlockFreq, PSI, &MBFIW->getMBFI(), QueryType);
+  return shouldOptimizeForSizeImpl(BlockFreq, PSI, &MBFIW->getMBFI(),
+                                   QueryType);
 }

llvm/lib/Transforms/Utils/SizeOpts.cpp

static bool isColdBlockNthPercentile(int CutOff, const BasicBlock *BB,
     return PSI->isColdBlockNthPercentile(CutOff, BB, BFI);
   }
 };
 } // end anonymous namespace
 
 bool llvm::shouldOptimizeForSize(const Function *F, ProfileSummaryInfo *PSI,
                                  BlockFrequencyInfo *BFI,
                                  PGSOQueryType QueryType) {
-  return shouldFuncOptimizeForSizeImpl<BasicBlockBFIAdapter>(F, PSI, BFI,
-                                                             QueryType);
+  return shouldFuncOptimizeForSizeImpl(F, PSI, BFI, QueryType);
 }
 
 bool llvm::shouldOptimizeForSize(const BasicBlock *BB, ProfileSummaryInfo *PSI,
                                  BlockFrequencyInfo *BFI,
                                  PGSOQueryType QueryType) {
   assert(BB);
-  return shouldOptimizeForSizeImpl<BasicBlockBFIAdapter>(BB, PSI, BFI,
-                                                         QueryType);
+  return shouldOptimizeForSizeImpl(BB, PSI, BFI, QueryType);
 }