Disable hoisting MI to hotter basic blocks

[NeHuang]

Summary

In current Hoist() function of machine licm pass, it will not check the source and destination basic block frequencies that a instruction is hoisted from/to. There is a chance that instruction is hoisted from a cold to a hot basic block.

In this patch, we add options to disable machine instruction hoisting if destination block is hotter.

[nemanjai]

Added a couple more reviewers that either recently modified this file or I think may be interested in the proposed change.

[NeHuang]

ping

[NeHuang]

ping

[efriedma]

In the future, please try to ping more frequently than once a month; we don't want to lose patches, even if there are some areas where it takes a while to get review.

The general idea of avoiding hoisting that would increase the frequency makes sense.

Could you give some idea what effect this has in practice? How often does this trigger on SPEC? Does it affect SPEC performance or size numbers? (Replace SPEC with some similar benchmark if you prefer.)

Is there anything special about the number "100"?

[NeHuang]

Thanks @efriedma for the review and remind.

This patch adds an option to disable non-profitable hoisting from a cold to a hot basic block.

The stats below is collected based on Google benchmark. (The 3rd column indicates number of hoisting is disabled due to block frequency hotness, results varies from the loads of different benchmarks)

Benchmark	Number of machine instructions hoisted out of loops	Number of instructions not hoisted due to block frequency hotness
Protobuf	20931	1706
Brotli	9935	17616
Gipfeli	471	879
snappy	886	1154
TCMalloc	6435	891
BoringSSL	6115	501
FFMPEG	8324	17

With the patch enabled, we found SPEC2017 performance gain for PPC.

'100' is the block-freq-ratio-threshold used in our experiment to determine when to disable the hoisting, which is a configurable option.

[Whitney]

llvm/lib/CodeGen/MachineLICM.cpp

54

Please run clang-format on your change. The include is not added at the right place.

1587

Change to ++ NumNotHoistedDueToHotness

[NeHuang]

This comment was removed by NeHuang.

llvm/lib/CodeGen/MachineLICM.cpp

54

Done.

1587

Done

[jsji]

machinelicm_clangformat_modv1.cpp ? I think you uploaded wrong patch.

[NeHuang]

@jsji Yes, please ignore diff 216684. Thanks you for the remind.

Thanks @Whitney for the review. Addressed the comment and uploaded diff 216693.

[lei]

llvm/lib/CodeGen/MachineLICM.cpp

113

nit: please follow the same indentation as the lines above.

113

indentation

1574

nit: Most of the functions here use either BB or MBB for MachineBasicBlock variables. IMHO unless there are diff BBs that are being tracked, I don't see why we need to specify this as "current"BB. Just makes it easier to read when similar variables are being used consistently.

[lebedev.ri]

Just passing-by remark: i find naming schemes in this patch very confusing:

• The patch subject should talk about not hoisting into hotter blocks, not about early out
• due to block hotness is very meaningless to me. I suspect most of the occurrences of that phrase should actually be closer to because destination block is hotter

llvm/lib/CodeGen/MachineLICM.cpp

102–103

It isn't really obvious whether larger value relaxes this check or the other way around

[NeHuang]

Thanks @lei for the review. Updated the patch to r

1. Resolve the indentation issues
2. Modify the variable name to be consistent with current naming convention.

Thanks @lebedev.ri for your review. Subject and summary have been updated accordingly.
For the BlockFrequencyRatioThreshold, larger value will relax the check. For example, by default, it will disable MI hoisting if destination block is 100 times hotter than source block.

[NeHuang]

llvm/lib/CodeGen/MachineLICM.cpp

113

Done

113

Done

[NeHuang]

llvm/lib/CodeGen/MachineLICM.cpp

1574

Done

[NeHuang]

llvm/lib/CodeGen/MachineLICM.cpp

1574

Done

[nemanjai]

Some minor changes are required to make it more clear what this does.

llvm/lib/CodeGen/MachineLICM.cpp

102–103

We should probably be more forthcoming with this description so that it is clear to the user.
Perhaps:

Do not hoist instructions if the target block is N times hotter than the source.

Also, we should add a comment here along the lines of:

// The default threshold of 100 (i.e. if target block is 100 times hotter)
// is based on empirical data on a single target and is subject to tuning.

106

Similarly to below, the naming is a bit strange. Perhaps:

enum DisableHoistingToHotterBlocks { None, PGO, All };

or

enum DisableHoistingToHotterBlocks { None, PGO_Only, Static };

(although I prefer the former).

109

The name sounds very strange. Perhaps:

DisableHoistingToHotterBlocks("disable-hoisting-to-hotter-blocks",

113

"block frequency hotness" sounds strange. Please use either "hotness" or "block frequency".

276

I think it is clearer to just use SrcBlock and TgtBlock for the names of the blocks.

1472

Presumably this function is called many times for the same function and we are querying the function for profile data every time. Would it be possible to just have a class member flag for whether the function has profile data or not?

1582

We don't want to increment the statistic here?

Honestly, I don't think we should be incrementing the statistic in this function at all - this is just a query for whether or not this is profitable. Strictly speaking, the increment to the statistic really belongs in the function that would actually do the hoisting (i.e. Hoist()).

1586

If we move the update to the statistic out of this function, this can probably just be changed to:
return Ratio > BlockFrequencyRatioThreshold;

[NeHuang]

Addressed @nemanjai 's review comment.

[NeHuang]

Thanks @nemanjai 's review. Addressed all the comments and updated the test cases accordingly.

llvm/lib/CodeGen/MachineLICM.cpp

106

• Using "DisableHositingToHotterBlocksType" for enum
• Using"NONE" instead of "None" to avoid ambiguity.

[NeHuang]

Collected stats below for SPECInt and SPECFP benchmarks (SPEC2017) with this feature enabled, compiled with -O3 and PGO.

Benchmark	Number of machine instructions hoisted out of loops	Number of instructions not hoisted due to block frequency
SPECInt	129269	459891
SPECFP	36047	70075

[lebedev.ri]

In D63676#1679799, @NeHuang wrote:

Collected stats below for SPECInt and SPECFP benchmarks (SPEC2017) with this feature enabled, compiled with -O3 and PGO.

Benchmark	Number of machine instructions hoisted out of loops	Number of instructions not hoisted due to block frequency
SPECInt	129269	459891
SPECFP	36047	70075

I think this might be a bit context-less, as in what are the numbers without the patch?

[NeHuang]

• Merged to latest code base and compiled with -O3 and PGO
• Collected stats for SPECInt and SPECFP benchmarks (SPEC2017) with baseline and patch.

Benchmark	Number of machine instructions hoisted out of loops (Baseline)	Number of machine instructions hoisted out of loops (Patch)	Number of instructions not hoisted to hotter destination (Patch)
SPECInt	206914	132007	476157
SPECFP	42591	36383	68215

• With the feature enabled, found performance gain for SPEC benchmarks, e.g. 2.1% for perlbench_r and 1.6% for povray_r.

[nemanjai]

With the minor comments addressed, this LGTM.
However before you update and commit, it would be good to hear from at least @lebedev.ri about whether his requests were satisfactorily met.

llvm/lib/CodeGen/MachineLICM.cpp

108

It is common practice to differentiate enumerators in one of two ways:

1. Prefix each enumerator with an acronym of the enumeration
2. Use scoped enumerations

While we're at it, you might as well simplify the naming to just UseBFI
Examples (with simplified name):

1. enum UseBFI { UseBFI_None, UseBFI_PGO, UseBFI_All }
2. enum class UseBFI { None, PGO, All }

My preference is 2.

112

Please replace all uses of the word hosting with the word hoisting since we are changing the decisions about hoisting instructions out of loops. In addition to correcting it in code, please correct it in the description, title, commit message, etc.

125

Nit: naming convention (variables still start with uppercase letters).

276

Nit: naming convention - functions start with a lowercase letter. And since this doesn't do any hoisting, I think a simpler name would be more appropriate. Perhaps

// Is the target basic block at least "BlockFrequencyRatioThreshold" times
// hotter than the source basic block.
bool isTgtHotterThanSrc(MachineBasicBlock *SrcBlock,
                        MachineBasicBlock *TgtBlock);

1575

This indentation is suspicious. If that's where clang-format put this line, so be it, but it just seems odd.

[NeHuang]

Addressed review comments from @nemanjai

@lebedev.ri I have re-collected stats for SPECInt and SPECFP benchmarks with baseline and patch. Table summarized above.

[NeHuang]

ping

[NeHuang]

Since the patch is approved, I will hold it for another 3 days if any comment and commit it on Monday.

[xbolva00]

This is disabled by default for non-PGO builds, right? Did you try to benchmark the patch for non PGO mode? There are various heuristic already in LLVM which are enabled for non-PGO mode too if they generally improve the perf.

[NeHuang]

Thanks @xbolva00 for your comment.

"This is disabled by default for non-PGO builds, right?"
Yes, it is disabled for non-pgo by default.

Did you try to benchmark the patch for non PGO mode? There are various heuristic already in LLVM which are enabled for non-PGO mode too if they generally improve the perf.
Yes, I did. For non-pgo mode, there are no significant improvements and one noticeable degradation for benchmark povray (4.7% degradation)

[xbolva00]

Ok, thanks for info!

llvm/lib/CodeGen/MachineLICM.cpp

 #include "llvm/ADT/BitVector.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/CodeGen/MachineBlockFrequencyInfo.h"
 #include "llvm/CodeGen/MachineDominators.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
 #include "llvm/CodeGen/MachineOperand.h"

 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cassert>
 #include <limits>
 #include <vector>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "machinelicm"
 
 static cl::opt<bool>
 AvoidSpeculation("avoid-speculation",
                  cl::desc("MachineLICM should avoid speculation"),
                  cl::init(true), cl::Hidden);
 
 static cl::opt<bool>
 HoistCheapInsts("hoist-cheap-insts",
                 cl::desc("MachineLICM should hoist even cheap instructions"),
                 cl::init(false), cl::Hidden);
 
 static cl::opt<bool>
 SinkInstsToAvoidSpills("sink-insts-to-avoid-spills",
                        cl::desc("MachineLICM should sink instructions into "
                                 "loops to avoid register spills"),
                        cl::init(false), cl::Hidden);
 static cl::opt<bool>
 HoistConstStores("hoist-const-stores",
                  cl::desc("Hoist invariant stores"),
                  cl::init(true), cl::Hidden);
+// The default threshold of 100 (i.e. if target block is 100 times hotter)
+// is based on empirical data on a single target and is subject to tuning.
+static cl::opt<unsigned>
+BlockFrequencyRatioThreshold("block-freq-ratio-threshold",
+                             cl::desc("Do not hoist instructions if target"
+                             "block is N times hotter than the source."),
+                             cl::init(100), cl::Hidden);
+
+enum class UseBFI { None, PGO, All };
+
+static cl::opt<UseBFI>
+DisableHoistingToHotterBlocks("disable-hoisting-to-hotter-blocks",
+                              cl::desc("Disable hoisting instructions to"
+                              " hotter blocks"),
+                              cl::init(UseBFI::None), cl::Hidden,
+                              cl::values(clEnumValN(UseBFI::None, "none",
+                              "disable the feature"),
+                              clEnumValN(UseBFI::PGO, "pgo",
+                              "enable the feature when using profile data"),
+                              clEnumValN(UseBFI::All, "all",
+                              "enable the feature with/wo profile data")));
 
 STATISTIC(NumHoisted,
           "Number of machine instructions hoisted out of loops");
 STATISTIC(NumLowRP,
           "Number of instructions hoisted in low reg pressure situation");
 STATISTIC(NumHighLatency,
           "Number of high latency instructions hoisted");
 STATISTIC(NumCSEed,
           "Number of hoisted machine instructions CSEed");
 STATISTIC(NumPostRAHoisted,
           "Number of machine instructions hoisted out of loops post regalloc");
 STATISTIC(NumStoreConst,
           "Number of stores of const phys reg hoisted out of loops");
+STATISTIC(NumNotHoistedDueToHotness,
+          "Number of instructions not hoisted due to block frequency");
 
 namespace {
 
   class MachineLICMBase : public MachineFunctionPass {
     const TargetInstrInfo *TII;
     const TargetLoweringBase *TLI;
     const TargetRegisterInfo *TRI;
     const MachineFrameInfo *MFI;
     MachineRegisterInfo *MRI;
     TargetSchedModel SchedModel;
     bool PreRegAlloc;
+    bool HasProfileData;
 
     // Various analyses that we use...
     AliasAnalysis        *AA;      // Alias analysis info.
+    MachineBlockFrequencyInfo *MBFI; // Machine block frequncy info
     MachineLoopInfo      *MLI;     // Current MachineLoopInfo
     MachineDominatorTree *DT;      // Machine dominator tree for the cur loop
 
     // State that is updated as we process loops
     bool         Changed;          // True if a loop is changed.
     bool         FirstInLoop;      // True if it's the first LICM in the loop.
     MachineLoop *CurLoop;          // The current loop we are working on.
     MachineBasicBlock *CurPreheader; // The preheader for CurLoop.
namespace {
   public:
     MachineLICMBase(char &PassID, bool PreRegAlloc)
         : MachineFunctionPass(PassID), PreRegAlloc(PreRegAlloc) {}
 
     bool runOnMachineFunction(MachineFunction &MF) override;
 
     void getAnalysisUsage(AnalysisUsage &AU) const override {
       AU.addRequired<MachineLoopInfo>();
+      if (DisableHoistingToHotterBlocks != UseBFI::None)
+        AU.addRequired<MachineBlockFrequencyInfo>();
       AU.addRequired<MachineDominatorTree>();
       AU.addRequired<AAResultsWrapperPass>();
       AU.addPreserved<MachineLoopInfo>();
       MachineFunctionPass::getAnalysisUsage(AU);
     }
 
     void releaseMemory() override {
       RegSeen.clear();
bool EliminateCSE(
         DenseMap<unsigned, std::vector<const MachineInstr *>>::iterator &CI);
 
     bool MayCSE(MachineInstr *MI);
 
     bool Hoist(MachineInstr *MI, MachineBasicBlock *Preheader);
 
     void InitCSEMap(MachineBasicBlock *BB);
 
+    bool isTgtHotterThanSrc(MachineBasicBlock *SrcBlock,
+                            MachineBasicBlock *TgtBlock);
     MachineBasicBlock *getCurPreheader();
   };
 
   class MachineLICM : public MachineLICMBase {
   public:
     static char ID;
     MachineLICM() : MachineLICMBase(ID, false) {
       initializeMachineLICMPass(*PassRegistry::getPassRegistry());

 char EarlyMachineLICM::ID;
 
 char &llvm::MachineLICMID = MachineLICM::ID;
 char &llvm::EarlyMachineLICMID = EarlyMachineLICM::ID;
 
 INITIALIZE_PASS_BEGIN(MachineLICM, DEBUG_TYPE,
                       "Machine Loop Invariant Code Motion", false, false)
 INITIALIZE_PASS_DEPENDENCY(MachineLoopInfo)
+INITIALIZE_PASS_DEPENDENCY(MachineBlockFrequencyInfo)
 INITIALIZE_PASS_DEPENDENCY(MachineDominatorTree)
 INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
 INITIALIZE_PASS_END(MachineLICM, DEBUG_TYPE,
                     "Machine Loop Invariant Code Motion", false, false)
 
 INITIALIZE_PASS_BEGIN(EarlyMachineLICM, "early-machinelicm",
                       "Early Machine Loop Invariant Code Motion", false, false)
 INITIALIZE_PASS_DEPENDENCY(MachineLoopInfo)
+INITIALIZE_PASS_DEPENDENCY(MachineBlockFrequencyInfo)
 INITIALIZE_PASS_DEPENDENCY(MachineDominatorTree)
 INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
 INITIALIZE_PASS_END(EarlyMachineLICM, "early-machinelicm",
                     "Early Machine Loop Invariant Code Motion", false, false)
 
 /// Test if the given loop is the outer-most loop that has a unique predecessor.
 static bool LoopIsOuterMostWithPredecessor(MachineLoop *CurLoop) {
   // Check whether this loop even has a unique predecessor.
bool MachineLICMBase::runOnMachineFunction(MachineFunction &MF) {
   TII = ST.getInstrInfo();
   TLI = ST.getTargetLowering();
   TRI = ST.getRegisterInfo();
   MFI = &MF.getFrameInfo();
   MRI = &MF.getRegInfo();
   SchedModel.init(&ST);
 
   PreRegAlloc = MRI->isSSA();
+  HasProfileData = MF.getFunction().hasProfileData();
 
   if (PreRegAlloc)
     LLVM_DEBUG(dbgs() << "******** Pre-regalloc Machine LICM: ");
   else
     LLVM_DEBUG(dbgs() << "******** Post-regalloc Machine LICM: ");
   LLVM_DEBUG(dbgs() << MF.getName() << " ********\n");
 
   if (PreRegAlloc) {
     // Estimate register pressure during pre-regalloc pass.
     unsigned NumRPS = TRI->getNumRegPressureSets();
     RegPressure.resize(NumRPS);
     std::fill(RegPressure.begin(), RegPressure.end(), 0);
     RegLimit.resize(NumRPS);
     for (unsigned i = 0, e = NumRPS; i != e; ++i)
       RegLimit[i] = TRI->getRegPressureSetLimit(MF, i);
   }
 
   // Get our Loop information...
+  if (DisableHoistingToHotterBlocks != UseBFI::None)
+    MBFI = &getAnalysis<MachineBlockFrequencyInfo>();
   MLI = &getAnalysis<MachineLoopInfo>();
   DT  = &getAnalysis<MachineDominatorTree>();
   AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
 
   SmallVector<MachineLoop *, 8> Worklist(MLI->begin(), MLI->end());
   while (!Worklist.empty()) {
     CurLoop = Worklist.pop_back_val();
     CurPreheader = nullptr;
bool MachineLICMBase::MayCSE(MachineInstr *MI) {
 
   return LookForDuplicate(MI, CI->second) != nullptr;
 }
 
 /// When an instruction is found to use only loop invariant operands
 /// that are safe to hoist, this instruction is called to do the dirty work.
 /// It returns true if the instruction is hoisted.
 bool MachineLICMBase::Hoist(MachineInstr *MI, MachineBasicBlock *Preheader) {
+  MachineBasicBlock *SrcBlock = MI->getParent();
+
+  // Disable the instruction hoisting due to block hotness
+  if ((DisableHoistingToHotterBlocks == UseBFI::All ||
+      (DisableHoistingToHotterBlocks == UseBFI::PGO && HasProfileData)) &&
+      isTgtHotterThanSrc(SrcBlock, Preheader)) {
+    ++NumNotHoistedDueToHotness;
+    return false;
+  }
   // First check whether we should hoist this instruction.
   if (!IsLoopInvariantInst(*MI) || !IsProfitableToHoist(*MI)) {
     // If not, try unfolding a hoistable load.
     MI = ExtractHoistableLoad(MI);
     if (!MI) return false;
   }
 
   // If we have hoisted an instruction that may store, it can only be a constant
if (!CurPreheader) {
       if (!CurPreheader) {
         CurPreheader = reinterpret_cast<MachineBasicBlock *>(-1);
         return nullptr;
       }
     }
   }
   return CurPreheader;
 }
+
+/// Is the target basic block at least "BlockFrequencyRatioThreshold"
+/// times hotter than the source basic block.
+bool MachineLICMBase::isTgtHotterThanSrc(MachineBasicBlock *SrcBlock,
+                                         MachineBasicBlock *TgtBlock) {
+  // Parse source and target basic block frequency from MBFI
+  uint64_t SrcBF = MBFI->getBlockFreq(SrcBlock).getFrequency();
+  uint64_t DstBF = MBFI->getBlockFreq(TgtBlock).getFrequency();
+
+  // Disable the hoisting if source block frequency is zero
+  if (!SrcBF)
+    return true;
+
+  double Ratio = (double)DstBF / SrcBF;
+
+  // Compare the block frequency ratio with the threshold
+  return Ratio > BlockFrequencyRatioThreshold;
+}

llvm/test/CodeGen/PowerPC/DisableHoistingDueToBlockHotnessNoProfileData.mir

+# NOTE: This test verifies disable/enable instruction hoisting to hot blocks based on non-profile data
+# RUN: llc -run-pass early-machinelicm -disable-hoisting-to-hotter-blocks=all -block-freq-ratio-threshold=100 %s -o - | FileCheck %s --check-prefix=CHECK-NO-HOIST
+# RUN: llc -run-pass early-machinelicm -disable-hoisting-to-hotter-blocks=all -block-freq-ratio-threshold=100000000 %s -o - | FileCheck %s --check-prefix=CHECK-HOIST
+# RUN: llc -run-pass early-machinelicm -disable-hoisting-to-hotter-blocks=pgo -block-freq-ratio-threshold=100 %s -o - | FileCheck %s --check-prefix=CHECK-HOIST
+# RUN: llc -run-pass early-machinelicm -disable-hoisting-to-hotter-blocks=none -block-freq-ratio-threshold=100 %s -o - | FileCheck %s --check-prefix=CHECK-HOIST
+
+--- |
+  target datalayout = "e-m:e-i64:64-n32:64"
+
+  define dso_local void @test(void (i32)* nocapture %fp, i32 signext %Arg, i32 signext %Len, i32* nocapture %Ptr) {
+  entry:
+    tail call void asm sideeffect "#NOTHING", "~{r2}"()
+    %cmp6 = icmp sgt i32 %Len, 0
+    br i1 %cmp6, label %for.body.lr.ph, label %for.cond.cleanup
+
+  for.body.lr.ph:                                   ; preds = %entry
+    %cmp1 = icmp sgt i32 %Arg, 10
+    br label %for.body
+
+  for.cond.cleanup:                                 ; preds = %for.inc, %entry
+    ret void
+
+  for.body:                                         ; preds = %for.inc, %for.body.lr.ph
+    %i.07 = phi i32 [ 0, %for.body.lr.ph ], [ %inc, %for.inc ]
+    %0 = load i32, i32* %Ptr, align 4
+    %1 = add i32 %i.07, %0
+    store i32 %1, i32* %Ptr, align 4
+    br i1 %cmp1, label %if.then, label %for.inc
+
+  if.then:                                          ; preds = %for.body
+    tail call void asm sideeffect "#NOTHING", "~{r2}"()
+    tail call void %fp(i32 signext %Arg)
+    br label %for.inc
+
+  for.inc:                                          ; preds = %if.then, %for.body
+    %inc = add nuw nsw i32 %i.07, 1
+    %exitcond = icmp eq i32 %Len, %inc
+    br i1 %exitcond, label %for.cond.cleanup, label %for.body
+  }
+
+  ; Function Attrs: nounwind
+  declare void @llvm.stackprotector(i8*, i8**) #0
+
+  attributes #0 = { nounwind }
+
+...
+---
+name:            test
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:
+  - { id: 0, class: crbitrc, preferred-register: '' }
+  - { id: 1, class: gprc_and_gprc_nor0, preferred-register: '' }
+  - { id: 2, class: gprc, preferred-register: '' }
+  - { id: 3, class: g8rc, preferred-register: '' }
+  - { id: 4, class: g8rc, preferred-register: '' }
+  - { id: 5, class: g8rc, preferred-register: '' }
+  - { id: 6, class: g8rc_and_g8rc_nox0, preferred-register: '' }
+  - { id: 7, class: gprc, preferred-register: '' }
+  - { id: 8, class: gprc, preferred-register: '' }
+  - { id: 9, class: crrc, preferred-register: '' }
+  - { id: 10, class: gprc, preferred-register: '' }
+  - { id: 11, class: crrc, preferred-register: '' }
+  - { id: 12, class: gprc, preferred-register: '' }
+  - { id: 13, class: gprc, preferred-register: '' }
+  - { id: 14, class: g8rc, preferred-register: '' }
+  - { id: 15, class: g8rc, preferred-register: '' }
+  - { id: 16, class: crrc, preferred-register: '' }
+liveins:
+  - { reg: '$x3', virtual-reg: '%3' }
+  - { reg: '$x4', virtual-reg: '%4' }
+  - { reg: '$x5', virtual-reg: '%5' }
+  - { reg: '$x6', virtual-reg: '%6' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    0
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x7ecade30), %bb.2(0x013521d0)
+    liveins: $x3, $x4, $x5, $x6
+
+    %6:g8rc_and_g8rc_nox0 = COPY $x6
+    %5:g8rc = COPY $x5
+    %4:g8rc = COPY $x4
+    %3:g8rc = COPY $x3
+    %7:gprc = COPY %4.sub_32
+    %8:gprc = COPY %5.sub_32
+    INLINEASM &"#NOTHING", 1, 12, implicit-def early-clobber $r2
+    %9:crrc = CMPWI %8, 1
+    BCC 12, killed %9, %bb.2
+    B %bb.1
+
+  bb.1.for.body.lr.ph:
+    successors: %bb.3(0x80000000)
+
+    %11:crrc = CMPWI %7, 10
+    %0:crbitrc = COPY %11.sub_gt
+    %10:gprc = LI 0
+    B %bb.3
+
+  bb.2.for.cond.cleanup:
+    BLR8 implicit $lr8, implicit $rm
+
+  bb.3.for.body:
+    successors: %bb.4(0x00000002), %bb.5(0x7ffffffe)
+
+    %1:gprc_and_gprc_nor0 = PHI %10, %bb.1, %2, %bb.5
+    %12:gprc = LWZ 0, %6 :: (load 4 from %ir.Ptr)
+    %13:gprc = ADD4 %1, killed %12
+    STW killed %13, 0, %6 :: (store 4 into %ir.Ptr)
+    BCn %0, %bb.5
+    B %bb.4
+
+  bb.4.if.then:
+    successors: %bb.5(0x80000000)
+
+    INLINEASM &"#NOTHING", 1, 12, implicit-def early-clobber $r2
+    ADJCALLSTACKDOWN 32, 0, implicit-def dead $r1, implicit $r1
+    %14:g8rc = COPY $x2
+    STD %14, 24, $x1 :: (store 8 into stack + 24)
+    %15:g8rc = EXTSW_32_64 %7
+    $x3 = COPY %15
+    $x12 = COPY %3
+    MTCTR8 %3, implicit-def $ctr8
+    BCTRL8_LDinto_toc 24, $x1, csr_svr464_altivec, implicit-def dead $lr8, implicit-def dead $x2, implicit $ctr8, implicit $rm, implicit $x3, implicit $x12, implicit $x2, implicit-def $r1
+    ADJCALLSTACKUP 32, 0, implicit-def dead $r1, implicit $r1
+
+  bb.5.for.inc:
+    successors: %bb.2(0x013521d0), %bb.3(0x7ecade30)
+
+    %2:gprc = nuw nsw ADDI %1, 1
+    %16:crrc = CMPLW %8, %2
+    BCC 76, killed %16, %bb.2
+    B %bb.3
+
+...
+
+# CHECK for enabling instruction hoisting
+#CHECK-LABEL: test
+#CHECK-HOIST: bb.1.for.body.lr.ph:
+#CHECK-HOIST: %14:g8rc = COPY $x2
+#CHECK-HOIST: STD %14, 24, $x1 :: (store 8 into stack + 24)
+#CHECK-HOIST: %15:g8rc = EXTSW_32_64 %7
+#CHECK-HOIST: B %bb.3
+
+#CHECK-HOIST: bb.4.if.then:
+#CHECK-HOIST-NOT: %14:g8rc = COPY $x2
+#CHECK-HOIST-NOT: STD %14, 24, $x1 :: (store 8 into stack + 24)
+#CHECK-HOIST-NOT: %15:g8rc = EXTSW_32_64 %7
+#CHECK-HOIST: bb.5.for.inc:
+
+# CHECK for disabling instruction hoisting due to block hotness
+#CHECK-LABEL: test
+#CHECK-NO-HOIST: bb.1.for.body.lr.ph:
+#CHECK-NO-HOIST-NOT: %14:g8rc = COPY $x2
+#CHECK-NO-HOIST-NOT: STD %14, 24, $x1 :: (store 8 into stack + 24)
+#CHECK-NO-HOIST-NOT: %15:g8rc = EXTSW_32_64 %7
+#CHECK-NO-HOIST: B %bb.3
+
+#CHECK-NO-HOIST: bb.4.if.then:
+#CHECK-NO-HOIST: %14:g8rc = COPY $x2
+#CHECK-NO-HOIST: STD %14, 24, $x1 :: (store 8 into stack + 24)
+#CHECK-NO-HOIST: %15:g8rc = EXTSW_32_64 %7
+#CHECK-NO-HOIST: bb.5.for.inc:

llvm/test/CodeGen/PowerPC/DisableHoistingDueToBlockHotnessProfileData.mir

+# NOTE: This test verifies disable/enable instruction hoisting to hot blocks based on profile data
+# RUN: llc -run-pass early-machinelicm -disable-hoisting-to-hotter-blocks=pgo -block-freq-ratio-threshold=100 %s -o - | FileCheck %s --check-prefix=CHECK-NO-HOIST
+# RUN: llc -run-pass early-machinelicm -disable-hoisting-to-hotter-blocks=pgo -block-freq-ratio-threshold=100000000 %s -o - | FileCheck %s --check-prefix=CHECK-HOIST
+# RUN: llc -run-pass early-machinelicm -disable-hoisting-to-hotter-blocks=none -block-freq-ratio-threshold=100 %s -o - | FileCheck %s --check-prefix=CHECK-HOIST
+
+--- |
+  target datalayout = "e-m:e-i64:64-n32:64"
+  target triple = "powerpc64le-unknown-linux-gnu"
+
+  ; Function Attrs: nounwind
+  define dso_local void @test(void (i32)* nocapture %fp, i32 signext %Arg, i32 signext %Len, i32* nocapture %Ptr) local_unnamed_addr #0 !prof !29 !section_prefix !30 {
+  entry:
+    tail call void asm sideeffect "#NOTHING", "~{r2}"() #1, !srcloc !31
+    %cmp6 = icmp sgt i32 %Len, 0
+    br i1 %cmp6, label %for.body.lr.ph, label %for.cond.cleanup, !prof !32
+
+  for.body.lr.ph:                                   ; preds = %entry
+    %cmp1 = icmp sgt i32 %Arg, 10
+    br label %for.body
+
+  for.cond.cleanup:                                 ; preds = %for.inc, %entry
+    ret void
+
+  for.body:                                         ; preds = %for.inc, %for.body.lr.ph
+    %i.07 = phi i32 [ 0, %for.body.lr.ph ], [ %inc, %for.inc ]
+    %0 = load i32, i32* %Ptr, align 4, !tbaa !33
+    %1 = add i32 %i.07, %0
+    store i32 %1, i32* %Ptr, align 4, !tbaa !33
+    br i1 %cmp1, label %if.then, label %for.inc, !prof !37
+
+  if.then:                                          ; preds = %for.body
+    tail call void asm sideeffect "#NOTHING", "~{r2}"() #1, !srcloc !31
+    tail call void %fp(i32 signext %Arg) #1, !prof !38
+    br label %for.inc
+
+  for.inc:                                          ; preds = %if.then, %for.body
+    %inc = add nuw nsw i32 %i.07, 1
+    %exitcond = icmp eq i32 %Len, %inc
+    br i1 %exitcond, label %for.cond.cleanup, label %for.body, !prof !39
+  }
+
+  ; Function Attrs: nounwind
+  declare void @llvm.stackprotector(i8*, i8**) #1
+
+  attributes #0 = { nounwind "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-frame-pointer-elim"="false" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="ppc64le" "target-features"="+altivec,+bpermd,+crypto,+direct-move,+extdiv,+htm,+power8-vector,+vsx,-power9-vector,-qpx" "unsafe-fp-math"="false" "use-soft-float"="false" }
+  attributes #1 = { nounwind }
+
+  !llvm.module.flags = !{!0, !1}
+  !llvm.ident = !{!28}
+
+  !0 = !{i32 1, !"wchar_size", i32 4}
+  !1 = !{i32 1, !"ProfileSummary", !2}
+  !2 = !{!3, !4, !5, !6, !7, !8, !9, !10}
+  !3 = !{!"ProfileFormat", !"InstrProf"}
+  !4 = !{!"TotalCount", i64 25405000087}
+  !5 = !{!"MaxCount", i64 21000000020}
+  !6 = !{!"MaxInternalCount", i64 200000003}
+  !7 = !{!"MaxFunctionCount", i64 21000000020}
+  !8 = !{!"NumCounts", i64 15}
+  !9 = !{!"NumFunctions", i64 7}
+  !10 = !{!"DetailedSummary", !11}
+  !11 = !{!12, !13, !14, !15, !16, !17, !18, !19, !20, !21, !22, !23, !24, !25, !26, !27}
+  !12 = !{i32 10000, i64 21000000020, i32 1}
+  !13 = !{i32 100000, i64 21000000020, i32 1}
+  !14 = !{i32 200000, i64 21000000020, i32 1}
+  !15 = !{i32 300000, i64 21000000020, i32 1}
+  !16 = !{i32 400000, i64 21000000020, i32 1}
+  !17 = !{i32 500000, i64 21000000020, i32 1}
+  !18 = !{i32 600000, i64 21000000020, i32 1}
+  !19 = !{i32 700000, i64 21000000020, i32 1}
+  !20 = !{i32 800000, i64 21000000020, i32 1}
+  !21 = !{i32 900000, i64 4203000000, i32 2}
+  !22 = !{i32 950000, i64 4203000000, i32 2}
+  !23 = !{i32 990000, i64 4203000000, i32 2}
+  !24 = !{i32 999000, i64 200000003, i32 3}
+  !25 = !{i32 999900, i64 200000003, i32 3}
+  !26 = !{i32 999990, i64 2000000, i32 4}
+  !27 = !{i32 999999, i64 2000000, i32 4}
+  !28 = !{!"clang version 9.0.0 ([e-mail]:compiler/llvm-project.git 01fc2fc8e690ee427cab149cb0bfd63568bed89b)"}
+  !29 = !{!"function_entry_count", i64 200000003}
+  !30 = !{!"function_section_prefix", !".hot"}
+  !31 = !{i32 65}
+  !32 = !{!"branch_weights", i32 -94967292, i32 40000000}
+  !33 = !{!34, !34, i64 0}
+  !34 = !{!"int", !35, i64 0}
+  !35 = !{!"omnipotent char", !36, i64 0}
+  !36 = !{!"Simple C/C++ TBAA"}
+  !37 = !{!"branch_weights", i32 4, i32 -94967296}
+  !38 = !{!"VP", i32 0, i64 20, i64 -3706093650706652785, i64 20}
+  !39 = !{!"branch_weights", i32 40000000, i32 -94967292}
+
+...
+---
+name:            test
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:
+  - { id: 0, class: crbitrc, preferred-register: '' }
+  - { id: 1, class: gprc_and_gprc_nor0, preferred-register: '' }
+  - { id: 2, class: gprc, preferred-register: '' }
+  - { id: 3, class: g8rc, preferred-register: '' }
+  - { id: 4, class: g8rc, preferred-register: '' }
+  - { id: 5, class: g8rc, preferred-register: '' }
+  - { id: 6, class: g8rc_and_g8rc_nox0, preferred-register: '' }
+  - { id: 7, class: gprc, preferred-register: '' }
+  - { id: 8, class: gprc, preferred-register: '' }
+  - { id: 9, class: crrc, preferred-register: '' }
+  - { id: 10, class: gprc, preferred-register: '' }
+  - { id: 11, class: crrc, preferred-register: '' }
+  - { id: 12, class: gprc, preferred-register: '' }
+  - { id: 13, class: gprc, preferred-register: '' }
+  - { id: 14, class: g8rc, preferred-register: '' }
+  - { id: 15, class: g8rc, preferred-register: '' }
+  - { id: 16, class: crrc, preferred-register: '' }
+liveins:
+  - { reg: '$x3', virtual-reg: '%3' }
+  - { reg: '$x4', virtual-reg: '%4' }
+  - { reg: '$x5', virtual-reg: '%5' }
+  - { reg: '$x6', virtual-reg: '%6' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    0
+  adjustsStack:    false
+  hasCalls:        true
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x7ecade30), %bb.2(0x013521d0)
+    liveins: $x3, $x4, $x5, $x6
+
+    %6:g8rc_and_g8rc_nox0 = COPY $x6
+    %5:g8rc = COPY $x5
+    %4:g8rc = COPY $x4
+    %3:g8rc = COPY $x3
+    %7:gprc = COPY %4.sub_32
+    %8:gprc = COPY %5.sub_32
+    INLINEASM &"#NOTHING", 1, 12, implicit-def early-clobber $r2, !31
+    %9:crrc = CMPWI %8, 1
+    BCC 12, killed %9, %bb.2
+    B %bb.1
+
+  bb.1.for.body.lr.ph:
+    successors: %bb.3(0x80000000)
+
+    %11:crrc = CMPWI %7, 10
+    %0:crbitrc = COPY %11.sub_gt
+    %10:gprc = LI 0
+    B %bb.3
+
+  bb.2.for.cond.cleanup:
+    BLR8 implicit $lr8, implicit $rm
+
+  bb.3.for.body:
+    successors: %bb.4(0x00000002), %bb.5(0x7ffffffe)
+
+    %1:gprc_and_gprc_nor0 = PHI %10, %bb.1, %2, %bb.5
+    %12:gprc = LWZ 0, %6 :: (load 4 from %ir.Ptr, !tbaa !33)
+    %13:gprc = ADD4 %1, killed %12
+    STW killed %13, 0, %6 :: (store 4 into %ir.Ptr, !tbaa !33)
+    BCn %0, %bb.5
+    B %bb.4
+
+  bb.4.if.then:
+    successors: %bb.5(0x80000000)
+
+    INLINEASM &"#NOTHING", 1, 12, implicit-def early-clobber $r2, !31
+    ADJCALLSTACKDOWN 32, 0, implicit-def dead $r1, implicit $r1
+    %14:g8rc = COPY $x2
+    STD %14, 24, $x1 :: (store 8 into stack + 24)
+    %15:g8rc = EXTSW_32_64 %7
+    $x3 = COPY %15
+    $x12 = COPY %3
+    MTCTR8 %3, implicit-def $ctr8
+    BCTRL8_LDinto_toc 24, $x1, csr_svr464_altivec, implicit-def dead $lr8, implicit-def dead $x2, implicit $ctr8, implicit $rm, implicit $x3, implicit $x12, implicit $x2, implicit-def $r1
+    ADJCALLSTACKUP 32, 0, implicit-def dead $r1, implicit $r1
+
+  bb.5.for.inc:
+    successors: %bb.2(0x013521d0), %bb.3(0x7ecade30)
+
+    %2:gprc = nuw nsw ADDI %1, 1
+    %16:crrc = CMPLW %8, %2
+    BCC 76, killed %16, %bb.2
+    B %bb.3
+
+...
+
+# CHECK for enabling instruction hoisting
+#CHECK-LABEL: test
+#CHECK-HOIST: bb.1.for.body.lr.ph:
+#CHECK-HOIST: %14:g8rc = COPY $x2
+#CHECK-HOIST: STD %14, 24, $x1 :: (store 8 into stack + 24)
+#CHECK-HOIST: %15:g8rc = EXTSW_32_64 %7
+#CHECK-HOIST: B %bb.3
+
+#CHECK-HOIST: bb.4.if.then:
+#CHECK-HOIST-NOT: %14:g8rc = COPY $x2
+#CHECK-HOIST-NOT: STD %14, 24, $x1 :: (store 8 into stack + 24)
+#CHECK-HOIST-NOT: %15:g8rc = EXTSW_32_64 %7
+#CHECK-HOIST: bb.5.for.inc:
+
+# CHECK for disabling instruction hoisting due to block hotness
+#CHECK-LABEL: test
+#CHECK-NO-HOIST: bb.1.for.body.lr.ph:
+#CHECK-NO-HOIST-NOT: %14:g8rc = COPY $x2
+#CHECK-NO-HOIST-NOT: STD %14, 24, $x1 :: (store 8 into stack + 24)
+#CHECK-NO-HOIST-NOT: %15:g8rc = EXTSW_32_64 %7
+#CHECK-NO-HOIST: B %bb.3
+
+#CHECK-NO-HOIST: bb.4.if.then:
+#CHECK-NO-HOIST: %14:g8rc = COPY $x2
+#CHECK-NO-HOIST: STD %14, 24, $x1 :: (store 8 into stack + 24)
+#CHECK-NO-HOIST: %15:g8rc = EXTSW_32_64 %7
+#CHECK-NO-HOIST: bb.5.for.inc:
+