[MLGO] Add BB Profile Dump Pass for Regalloc Case

[aidengrossman]

Summary

This patch adds a basic block profile dump pass that runs right before
asmprint time and dumps basic block profile information so that cost
models can use this data during downstream analysis.

[aidengrossman]

Pinging Rahman (at the request of Mircea) since this is supposed to integrate with the basic block section labels infrastructure currently being used for Propeller.

Currently keeping a raw_fd_ostream in the MachineFunctionPass. They're not supposed to keep state, so I'm not sure this is entirely idiomatic, but I didn't see any other obvious way to do it (might be missing something of course). Other than that should be good to go.

[aidengrossman]

Move profile dump option out of LLVM_HAVE_TFLITE gate to prevent compilation
failure if building without TFLite.

[mtrofin]

You'll need to update the failing pass structure tests, when you do, can you put a comment there (in the tests) that this pass must be right before asmprinter to make sure nothing mutates MBB freqs?

llvm/include/llvm/CodeGen/Passes.h

582

It's not really regalloc-tied, maybe createFinalProfileDumpPass (so "FinalProfileDump" being the pass)?

llvm/lib/CodeGen/MLRegallocEvictAdvisor.cpp

56 ↗

(On Diff #494795)

not regalloc. Maybe mbb-profile-dump?

Also, can the comment say that matching this info with BBs afterwards, using e.g. llvm-objdump --symbolize-operands, requires the compilation be performed with -fbasic-block-sections=labels?

122 ↗

(On Diff #494795)

needs its own .cpp, it's general-purpose

1190 ↗

(On Diff #494795)

@rahmanl this is where we wanna make sure we get it right: is this the right API that would ensure the output obtained here (so right before AsmPrint) matches the BB #s you get if you -fbasic-block-sections=lables and then llvm-objdump --symbolize-operands - I realize the latter outputs BB<nr>, but other than that (i.e. ignoring the "BB" prefix)

[rahmanl]

llvm/lib/CodeGen/MLRegallocEvictAdvisor.cpp

1190 ↗

(On Diff #494795)

The BB symbols printed by llvm-objdump will match getNumber() here as long as this pass comes right before AsmPrinter and right after calls to RenumberBlocks() .-fbasic-block-sections will do this for you. But if you don't want to use the flag, you can renumber here.

[aidengrossman]

Rename pass to MBBProfileDump and split into a separate file from the
regalloc eviction advisor. Modifying pipeline tests is still TODO.

[aidengrossman]

Update comment in TargetPassConfig.cpp

llvm/include/llvm/CodeGen/Passes.h

582

I renamed the pass to MBBProfileDump and changed all the associated init/creation functions. Also moved all the code into MBBProfileDump.cpp.

[rahmanl]

llvm/lib/CodeGen/MBBProfileDump.cpp

75

This won't be pretty in Thin-LTO mode which can spawn multiple threads to perform codegen across various modules. Is this intentional?

[mtrofin]

llvm/lib/CodeGen/MBBProfileDump.cpp

27

anonymous namespace; then just define outside that, fully qualified, the createXYZPass function (i.e. llvm::createMBBProfileDumpPass) see llvm/lib/Target/X86/X86DiscriminateMemOps.cpp for example.

74

nit: const auto &MBB

[aidengrossman]

Add note in command line option help specifying that this option does not
support ThinLTO. Switch from putting everything in the LLVM namespace to
using an anonymous namespace.

[aidengrossman]

llvm/lib/CodeGen/MBBProfileDump.cpp

75

Yes. We currently aren't supporting ThinLTO as when working with the MLGO infrastructure (where this is intended to fit in), ThinLTO is done at a previous stage during corpus collection where we won't be specifying this flag. I've added a note into the CLI flag help to note that it's not currently supported with ThinLTO.

[aidengrossman]

Address comments, fix pipeline tests that changed after the new pass addition,
and switched from requiring a MachineBlockFrequencyInfo analysis to a
LazyMachineBlockFrequencyInfo Pass as it calls MBFI under the hood, is already
called at the end of the current pass pipeline on targets, and prevents extra
work from being done in the case no profile dump is desired (ie the flag is
not set).

I have not added comments to any of the pipeline passes since they seem to be
at least partially autogenerted and the dependency/timing info (that this
pass needs to be right before ASMPrinter time) is already available in the
target pass config. Will change if still desired however.

[aidengrossman]

llvm/lib/CodeGen/MBBProfileDump.cpp

75

Clarification: We aren't supporting in-process (ie multithreaded single process thinLTO) in the same compiler invocation with this flag, but all thinLTO cases should work with corpus collection, and distributed thinLTO should work as well since everything is per compiler invocation.

[mtrofin]

llvm/lib/CodeGen/MBBProfileDump.cpp

26

The sentence ends mid air, how about "Enabling this flag during in-process ThinLTO is not supported."

llvm/test/CodeGen/AMDGPU/llc-pipeline.ll

719

Here and below for AMDGPU, it seems like the MBB Profile Dump should run in the next FunctionPass Manager.

[aidengrossman]

Fix CLI flag help comment based on reviewer feedback.

[aidengrossman]

Move infrastructure from separate MachineFunctionPass to within AsmPrinter.
This should reduce maintenance, make it less likely that the pass ever gets
moved around in the pipeline in a manner that causes it to cease to function,
and doesn't need modificatio of the pass pipeline tests.

[aidengrossman]

Update commit message to better reflect recent changes.

[mtrofin]

Thanks for doing this, looks so much simpler - basically ready to go imho after addressing the error handling comment

llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp

363 ↗

(On Diff #495722)

do this in doInitialize() because then you have a LLVMContext which you can use to emitError instead of errs()

377 ↗

(On Diff #495722)

well, for symmetry, do this in doFinalize. Although IIUC the dtor of raw_fd_ostream would close, too, so you could just skip this.

1889 ↗

(On Diff #495722)

nit: maybe test on if MBBProfileDumpFileOutput instead.

[aidengrossman]

Update based on reviewer feedback.

[thakis]

The enable of this test without tflite broke tests on Mac: http://45.33.8.238/macm1/54722/step_11.txt

Please take a look and revert for now if it takes a while to fix.

(Saying this here since the other change doesn't have a phab link in its commit message.)

[mtrofin]

In D143311#4120232, @thakis wrote:

The enable of this test without tflite broke tests on Mac: http://45.33.8.238/macm1/54722/step_11.txt

Please take a look and revert for now if it takes a while to fix.

(Saying this here since the other change doesn't have a phab link in its commit message.)

I'll send a patch in a moment. It's probably because the .ll is missing a triple. The test has no dependency on tflite, nor is it regalloc specific (albeit motivated by a regalloc and a mlgo training usecase).

[mtrofin]

In D143311#4120232, @thakis wrote:

The enable of this test without tflite broke tests on Mac: http://45.33.8.238/macm1/54722/step_11.txt

Please take a look and revert for now if it takes a while to fix.

(Saying this here since the other change doesn't have a phab link in its commit message.)

@thakis can you confirm D143815 - on my mac (x86), the test passed even before the proposed patch, and I'm not able to glean from your bot what (if anything) could be different between my setup and yours - I looked into http://45.33.8.238/macm1/54722/log.txt and http://45.33.8.238/macm1/54722/, looking for a gn or gn config.

My setup is as vanilla as cmake -GNinja ../llvm -DCMAKE_BUILD_TYPE=Release

llvm/include/llvm/CodeGen/Passes.h

namespace llvm {
   /// The pass transforms amx intrinsics to scalar operation if the function has
   /// optnone attribute or it is O0.
   FunctionPass *createX86LowerAMXIntrinsicsPass();
 
   /// When learning an eviction policy, extract score(reward) information,
   /// otherwise this does nothing
   FunctionPass *createRegAllocScoringPass();
 
+  // Export machine basic block profiles to a file if the necessary command
+  // line flags are set.
+  FunctionPass *createMBBProfileDumpPass();
+
   /// JMC instrument pass.
   ModulePass *createJMCInstrumenterPass();
 
   /// This pass converts conditional moves to conditional jumps when profitable.
   FunctionPass *createSelectOptimizePass();
+
+  // This pass exports basic block profile infromation for downstream cost
+  // modelling applications.
+  FunctionPass *createMBBProfileDumpPass();
 } // End llvm namespace
 
 #endif

llvm/include/llvm/InitializePasses.h

 void initializeRAGreedyPass(PassRegistry&);
 void initializeReachingDefAnalysisPass(PassRegistry&);
 void initializeReassociateLegacyPassPass(PassRegistry&);
 void initializeRedundantDbgInstEliminationPass(PassRegistry&);
 void initializeRegAllocEvictionAdvisorAnalysisPass(PassRegistry &);
 void initializeRegAllocFastPass(PassRegistry&);
 void initializeRegAllocPriorityAdvisorAnalysisPass(PassRegistry &);
 void initializeRegAllocScoringPass(PassRegistry &);
+void initializeMBBProfileDumpPass(PassRegistry &);
 void initializeRegBankSelectPass(PassRegistry&);
 void initializeRegToMemLegacyPass(PassRegistry&);
 void initializeRegUsageInfoCollectorPass(PassRegistry&);
 void initializeRegUsageInfoPropagationPass(PassRegistry&);
 void initializeRegionInfoPassPass(PassRegistry&);
 void initializeRegionOnlyPrinterPass(PassRegistry&);
 void initializeRegionOnlyViewerPass(PassRegistry&);
 void initializeRegionPrinterPass(PassRegistry&);

llvm/lib/CodeGen/CMakeLists.txt

add_llvm_component_library(LLVMCodeGen
   MachineSink.cpp
   MachineSizeOpts.cpp
   MachineSSAContext.cpp
   MachineSSAUpdater.cpp
   MachineStripDebug.cpp
   MachineTraceMetrics.cpp
   MachineUniformityAnalysis.cpp
   MachineVerifier.cpp
+  MBBProfileDump.cpp
   MIRFSDiscriminator.cpp
   MIRSampleProfile.cpp
   MIRYamlMapping.cpp
   MLRegallocEvictAdvisor.cpp
   MLRegallocPriorityAdvisor.cpp
   ModuloSchedule.cpp
   MultiHazardRecognizer.cpp
   PatchableFunction.cpp

llvm/lib/CodeGen/MBBProfileDump.cpp

+//===- MBBProfileDump.cpp - MBB Profile Dump Pass -------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Implementation of the MBB Profile Dump Pass
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/CodeGen/LazyMachineBlockFrequencyInfo.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Support/CommandLine.h"
+
+using namespace llvm;
+
+static cl::opt<std::string> BasicBlockProfileDump(
+    "mbb-profile-dump", cl::Hidden,
+    cl::desc("Basic block profile dump for external cost modelling. If "
+             "matching up BBs with afterwards, the compilation must be "
+             "performed with -fbasic-block-sections=labels. Enabling this "
+             "flag during in-process ThinLTO is not supported."));
+
+namespace {
+class MBBProfileDump : public MachineFunctionPass {
+public:
+  static char ID;
+  std::unique_ptr<raw_fd_ostream> FileOutput;
+
+  MBBProfileDump() : MachineFunctionPass(ID) {
+    initializeMBBProfileDumpPass(*PassRegistry::getPassRegistry());
+    if (!BasicBlockProfileDump.empty()) {
+      std::error_code PossibleFileError;
+      FileOutput = std::make_unique<raw_fd_ostream>(BasicBlockProfileDump,
+                                                    PossibleFileError);
+    }
+  }
+
+  ~MBBProfileDump() override {
+    if (FileOutput) {
+      FileOutput->close();
+    }
+  }
+
+  StringRef getPassName() const override { return "MBB Profile Dump"; }
+
+  // Analysis usage
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesAll();
+    AU.addRequired<LazyMachineBlockFrequencyInfoPass>();
+    MachineFunctionPass::getAnalysisUsage(AU);
+  }
+
+  // Perform the pass
+  bool runOnMachineFunction(MachineFunction &) override;
+};
+
+} // anonymous namespace
+
+char MBBProfileDump::ID = 0;
+FunctionPass *llvm::createMBBProfileDumpPass() { return new MBBProfileDump(); }
+
+INITIALIZE_PASS(MBBProfileDump, "mbbprofiledump", "MBB Profile Dump", false,
+                false)
+
+bool MBBProfileDump::runOnMachineFunction(MachineFunction &MF) {
+  if (!BasicBlockProfileDump.empty()) {
+    MachineBlockFrequencyInfo &MBFI =
+        getAnalysis<LazyMachineBlockFrequencyInfoPass>().getBFI();
+    for (const auto &MBB : MF) {
+      *FileOutput.get() << MF.getName() << "," << MBB.getNumber() << ","
+                        << MBFI.getBlockFreqRelativeToEntryBlock(&MBB) << "\n";
+    }
+  }
+  return false;
+}

llvm/lib/CodeGen/TargetPassConfig.cpp

void TargetPassConfig::addMachinePasses() {
   }
 
   if (!DisableCFIFixup && TM->Options.EnableCFIFixup)
     addPass(createCFIFixup());
 
   // Add passes that directly emit MI after all other MI passes.
   addPreEmitPass2();
 
+  // Export machine basic block profile information to be used in downstream
+  // cost modelling applications. This needs to be right before the AsmPrinter
+  // so that nothing following it will mutate MBB numbers or frequencies.
+  addPass(createMBBProfileDumpPass());
+
   AddingMachinePasses = false;
 }
 
 /// Add passes that optimize machine instructions in SSA form.
 void TargetPassConfig::addMachineSSAOptimization() {
   // Pre-ra tail duplication.
   addPass(&EarlyTailDuplicateID);
 

llvm/test/CodeGen/AArch64/O0-pipeline.ll

 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       Insert CFI remember/restore state instructions
 ; CHECK-NEXT:       Unpack machine instruction bundles
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
+; CHECK-NEXT:       MBB Profile Dump
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       AArch64 Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction
 
 define void @f() {
   ret void
 }

llvm/test/CodeGen/AArch64/O3-pipeline.ll

 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:     Machine Outliner
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Insert CFI remember/restore state instructions
 ; CHECK-NEXT:       Unpack machine instruction bundles
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
+; CHECK-NEXT:       MBB Profile Dump
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       AArch64 Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction
 ; CHECK-NEXT: Pass Arguments:  -domtree
 ; CHECK-NEXT:   FunctionPass Manager
 ; CHECK-NEXT:     Dominator Tree Construction
 ; CHECK-NEXT: Pass Arguments:  -assumption-cache-tracker -targetlibinfo -domtree -loops -scalar-evolution -stack-safety-local
 ; CHECK-NEXT: Assumption Cache Tracker

llvm/test/CodeGen/AArch64/arm64-opt-remarks-lazy-bfi.ll

 ; HOTNESS-NEXT: Freeing Pass 'Insert CFI remember/restore state instructions' on Function 'empty_func'
 ; HOTNESS-NEXT: Executing Pass 'Verify generated machine code'
 ; HOTNESS-NEXT: Freeing Pass 'Verify generated machine code'
 ; HOTNESS-NEXT: Executing Pass 'Unpack machine instruction bundles'
 ; HOTNESS-NEXT: Freeing Pass 'Unpack machine instruction bundles'
 ; HOTNESS-NEXT: Executing Pass 'Verify generated machine code'
 ; HOTNESS-NEXT: Freeing Pass 'Verify generated machine code'
 ; HOTNESS-NEXT: Executing Pass 'Lazy Machine Block Frequency Analysis'
+; HOTNESS-NEXT: Executing Pass 'MBB Profile Dump'
+; HOTNESS-NEXT: Freeing Pass 'MBB Profile Dump'
+; HOTNESS-NEXT: Executing Pass 'Verify generated machine code'
+; HOTNESS-NEXT: Freeing Pass 'Verify generated machine code'
 ; HOTNESS-NEXT: Executing Pass 'Machine Optimization Remark Emitter'
 ; HOTNESS-NEXT: Building MachineBlockFrequencyInfo on the fly
 ; HOTNESS-NEXT: Building LoopInfo on the fly
 ; HOTNESS-NEXT: Building DominatorTree on the fly
 ; HOTNESS-NOT: Executing Pass
 ; HOTNESS: block-frequency: empty_func
 ; HOTNESS-NOT: Executing Pass
 ; HOTNESS: Executing Pass 'AArch64 Assembly Printer'

 ; NO_HOTNESS-NEXT: Freeing Pass 'Insert CFI remember/restore state instructions' on Function 'empty_func'
 ; NO_HOTNESS-NEXT: Executing Pass 'Verify generated machine code'
 ; NO_HOTNESS-NEXT: Freeing Pass 'Verify generated machine code'
 ; NO_HOTNESS-NEXT: Executing Pass 'Unpack machine instruction bundles'
 ; NO_HOTNESS-NEXT: Freeing Pass 'Unpack machine instruction bundles'
 ; NO_HOTNESS-NEXT: Executing Pass 'Verify generated machine code'
 ; NO_HOTNESS-NEXT: Freeing Pass 'Verify generated machine code'
 ; NO_HOTNESS-NEXT: Executing Pass 'Lazy Machine Block Frequency Analysis'
+; NO_HOTNESS-NEXT: Executing Pass 'MBB Profile Dump'
+; NO_HOTNESS-NEXT: Freeing Pass 'MBB Profile Dump'
+; NO_HOTNESS-NEXT: Executing Pass 'Verify generated machine code'
+; NO_HOTNESS-NEXT: Freeing Pass 'Verify generated machine code'
 ; NO_HOTNESS-NEXT: Executing Pass 'Machine Optimization Remark Emitter'
 ; NO_HOTNESS-NEXT: Executing Pass 'AArch64 Assembly Printer'
 
 ; NO_HOTNESS: arm64-summary-remarks.ll:5:0: 1 instructions in function{{$}}
 
 define void @empty_func() nounwind ssp !dbg !3 !prof !4 {
   ret void
 }

llvm/test/CodeGen/AMDGPU/llc-pipeline.ll

 ; GCN-O0-NEXT:        SI insert wait instructions
 ; GCN-O0-NEXT:        Insert required mode register values
 ; GCN-O0-NEXT:        SI Final Branch Preparation
 ; GCN-O0-NEXT:        Post RA hazard recognizer
 ; GCN-O0-NEXT:        Branch relaxation pass
 ; GCN-O0-NEXT:        Register Usage Information Collector Pass
 ; GCN-O0-NEXT:        Live DEBUG_VALUE analysis
 ; GCN-O0-NEXT:        Machine Sanitizer Binary Metadata
+; GCN-O0-NEXT:        Lazy Machine Block Frequency Analysis
+; GCN-O0-NEXT:        MBB Profile Dump
 ; GCN-O0-NEXT:    Function register usage analysis
 ; GCN-O0-NEXT:    FunctionPass Manager
 ; GCN-O0-NEXT:      Lazy Machine Block Frequency Analysis
 ; GCN-O0-NEXT:      Machine Optimization Remark Emitter
 ; GCN-O0-NEXT:      AMDGPU Assembly Printer
 ; GCN-O0-NEXT:      Free MachineFunction
 ; GCN-O0-NEXT:Pass Arguments:  -domtree
 ; GCN-O0-NEXT:  FunctionPass Manager

 ; GCN-O1-NEXT:        SI Final Branch Preparation
 ; GCN-O1-NEXT:        SI peephole optimizations
 ; GCN-O1-NEXT:        Post RA hazard recognizer
 ; GCN-O1-NEXT:        AMDGPU Insert Delay ALU
 ; GCN-O1-NEXT:        Branch relaxation pass
 ; GCN-O1-NEXT:        Register Usage Information Collector Pass
 ; GCN-O1-NEXT:        Live DEBUG_VALUE analysis
 ; GCN-O1-NEXT:        Machine Sanitizer Binary Metadata
+; GCN-O1-NEXT:        Lazy Machine Block Frequency Analysis
+; GCN-O1-NEXT:        MBB Profile Dump
 ; GCN-O1-NEXT:    Function register usage analysis
 ; GCN-O1-NEXT:    FunctionPass Manager
 ; GCN-O1-NEXT:      Lazy Machine Block Frequency Analysis
 ; GCN-O1-NEXT:      Machine Optimization Remark Emitter
 ; GCN-O1-NEXT:      AMDGPU Assembly Printer
 ; GCN-O1-NEXT:      Free MachineFunction
 ; GCN-O1-NEXT:Pass Arguments:  -domtree
 ; GCN-O1-NEXT:  FunctionPass Manager

 ; GCN-O1-OPTS-NEXT:        SI Final Branch Preparation
 ; GCN-O1-OPTS-NEXT:        SI peephole optimizations
 ; GCN-O1-OPTS-NEXT:        Post RA hazard recognizer
 ; GCN-O1-OPTS-NEXT:        AMDGPU Insert Delay ALU
 ; GCN-O1-OPTS-NEXT:        Branch relaxation pass
 ; GCN-O1-OPTS-NEXT:        Register Usage Information Collector Pass
 ; GCN-O1-OPTS-NEXT:        Live DEBUG_VALUE analysis
 ; GCN-O1-OPTS-NEXT:        Machine Sanitizer Binary Metadata
+; GCN-O1-OPTS-NEXT:        Lazy Machine Block Frequency Analysis
+; GCN-O1-OPTS-NEXT:        MBB Profile Dump
 ; GCN-O1-OPTS-NEXT:    Function register usage analysis
 ; GCN-O1-OPTS-NEXT:    FunctionPass Manager
 ; GCN-O1-OPTS-NEXT:      Lazy Machine Block Frequency Analysis
 ; GCN-O1-OPTS-NEXT:      Machine Optimization Remark Emitter
 ; GCN-O1-OPTS-NEXT:      AMDGPU Assembly Printer
 ; GCN-O1-OPTS-NEXT:      Free MachineFunction
 ; GCN-O1-OPTS-NEXT:Pass Arguments:  -domtree
 ; GCN-O1-OPTS-NEXT:  FunctionPass Manager

 ; GCN-O2-NEXT:        SI peephole optimizations
 ; GCN-O2-NEXT:        Post RA hazard recognizer
 ; GCN-O2-NEXT:        Release VGPRs
 ; GCN-O2-NEXT:        AMDGPU Insert Delay ALU
 ; GCN-O2-NEXT:        Branch relaxation pass
 ; GCN-O2-NEXT:        Register Usage Information Collector Pass
 ; GCN-O2-NEXT:        Live DEBUG_VALUE analysis
 ; GCN-O2-NEXT:        Machine Sanitizer Binary Metadata
+; GCN-O2-NEXT:        Lazy Machine Block Frequency Analysis
+; GCN-O2-NEXT:        MBB Profile Dump
 ; GCN-O2-NEXT:    Function register usage analysis
 ; GCN-O2-NEXT:    FunctionPass Manager
 ; GCN-O2-NEXT:      Lazy Machine Block Frequency Analysis
 ; GCN-O2-NEXT:      Machine Optimization Remark Emitter
 ; GCN-O2-NEXT:      AMDGPU Assembly Printer
 ; GCN-O2-NEXT:      Free MachineFunction
 ; GCN-O2-NEXT:Pass Arguments:  -domtree
 ; GCN-O2-NEXT:  FunctionPass Manager

 ; GCN-O3-NEXT:        SI peephole optimizations
 ; GCN-O3-NEXT:        Post RA hazard recognizer
 ; GCN-O3-NEXT:        Release VGPRs
 ; GCN-O3-NEXT:        AMDGPU Insert Delay ALU
 ; GCN-O3-NEXT:        Branch relaxation pass
 ; GCN-O3-NEXT:        Register Usage Information Collector Pass
 ; GCN-O3-NEXT:        Live DEBUG_VALUE analysis
 ; GCN-O3-NEXT:        Machine Sanitizer Binary Metadata
+; GCN-O3-NEXT:        Lazy Machine Block Frequency Analysis
+; GCN-O3-NEXT:        MBB Profile Dump
 ; GCN-O3-NEXT:    Function register usage analysis
 ; GCN-O3-NEXT:    FunctionPass Manager
 ; GCN-O3-NEXT:      Lazy Machine Block Frequency Analysis
 ; GCN-O3-NEXT:      Machine Optimization Remark Emitter
 ; GCN-O3-NEXT:      AMDGPU Assembly Printer
 ; GCN-O3-NEXT:      Free MachineFunction
 ; GCN-O3-NEXT:Pass Arguments:  -domtree
 ; GCN-O3-NEXT:  FunctionPass Manager
 ; GCN-O3-NEXT:    Dominator Tree Construction
 
 define void @empty() {
   ret void
 }

llvm/test/CodeGen/ARM/O3-pipeline.ll

 ; CHECK-NEXT:      ARM Branch Targets
 ; CHECK-NEXT:      MachineDominator Tree Construction
 ; CHECK-NEXT:      ARM constant island placement and branch shortening pass
 ; CHECK-NEXT:      MachineDominator Tree Construction
 ; CHECK-NEXT:      Machine Natural Loop Construction
 ; CHECK-NEXT:      ReachingDefAnalysis
 ; CHECK-NEXT:      ARM Low Overhead Loops pass
 ; CHECK-NEXT:      Lazy Machine Block Frequency Analysis
+; CHECK-NEXT:      MBB Profile Dump
 ; CHECK-NEXT:      Machine Optimization Remark Emitter
 ; CHECK-NEXT:      ARM Assembly Printer
 ; CHECK-NEXT:      Free MachineFunction

llvm/test/CodeGen/LoongArch/O0-pipeline.ll

 ; CHECK-NEXT:       Implement the 'patchable-function' attribute
 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       LoongArch atomic pseudo instruction expansion pass
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
+; CHECK-NEXT:       MBB Profile Dump
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       LoongArch Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/LoongArch/opt-pipeline.ll

 ; CHECK-NEXT:       Implement the 'patchable-function' attribute
 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       LoongArch atomic pseudo instruction expansion pass
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
+; CHECK-NEXT:       MBB Profile Dump
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       LoongArch Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/MLRegalloc/bb-profile-dump.ll

+; REQUIRES: have_tflite
+; REQUIRES: default_triple
+;
+; Check that the basic block profile dump outputs data and in the correct
+; format.
+;
+; RUN: llc -o /dev/null -mbb-profile-dump=%t %s
+; RUN: FileCheck --input-file %t %s
+
+define i64 @f2(i64 %a, i64 %b) {
+    %sum = add i64 %a, %b
+    ret i64 %sum
+}
+
+define i64 @f1() {
+    %sum = call i64 @f2(i64 2, i64 2)
+    ret i64 %sum
+}
+
+; CHECK: f2,0,1.000000e+00
+; CHECK-NEXT: f1,0,1.000000e+00

llvm/test/CodeGen/PowerPC/O0-pipeline.ll

 ; CHECK-NEXT:       PowerPC Expand ISEL Generation
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       PowerPC Expand Atomic
 ; CHECK-NEXT:       PowerPC Branch Selector
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
+; CHECK-NEXT:       MBB Profile Dump
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Linux PPC Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction
 
 define void @f() {
   ret void
 }

llvm/test/CodeGen/PowerPC/O3-pipeline.ll

 ; CHECK-NEXT:       PowerPC Early-Return Creation
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       PowerPC Expand Atomic
 ; CHECK-NEXT:       PowerPC Branch Selector
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
+; CHECK-NEXT:       MBB Profile Dump
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Linux PPC Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction
 
 define void @f() {
   ret void
 }

llvm/test/CodeGen/RISCV/O0-pipeline.ll

 ; CHECK-NEXT:       RISCV Make Compressible
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       RISCV pseudo instruction expansion pass
 ; CHECK-NEXT:       RISCV atomic pseudo instruction expansion pass
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
+; CHECK-NEXT:       MBB Profile Dump
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       RISCV Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/RISCV/O3-pipeline.ll

 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:     Machine Outliner
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       RISCV pseudo instruction expansion pass
 ; CHECK-NEXT:       RISCV atomic pseudo instruction expansion pass
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
+; CHECK-NEXT:       MBB Profile Dump
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       RISCV Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/X86/O0-pipeline.ll

 ; CHECK-NEXT:       X86 Speculative Execution Side Effect Suppression
 ; CHECK-NEXT:       X86 Indirect Thunks
 ; CHECK-NEXT:       X86 Return Thunks
 ; CHECK-NEXT:       Check CFA info and insert CFI instructions if needed
 ; CHECK-NEXT:       X86 Load Value Injection (LVI) Ret-Hardening
 ; CHECK-NEXT:       Pseudo Probe Inserter
 ; CHECK-NEXT:       Unpack machine instruction bundles
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
+; CHECK-NEXT:       MBB Profile Dump
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       X86 Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction
 
 define void @f() {
   ret void
 }

llvm/test/CodeGen/X86/opt-pipeline.ll

 ; CHECK-NEXT:       X86 Speculative Execution Side Effect Suppression
 ; CHECK-NEXT:       X86 Indirect Thunks
 ; CHECK-NEXT:       X86 Return Thunks
 ; CHECK-NEXT:       Check CFA info and insert CFI instructions if needed
 ; CHECK-NEXT:       X86 Load Value Injection (LVI) Ret-Hardening
 ; CHECK-NEXT:       Pseudo Probe Inserter
 ; CHECK-NEXT:       Unpack machine instruction bundles
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
+; CHECK-NEXT:       MBB Profile Dump
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       X86 Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction
 
 ; We should only have one function pass manager.
 ; In the past, module passes have accidentally been added into the middle of
 ; the codegen pipeline, implicitly creating new function pass managers.
 ; FPM: FunctionPass Manager
 ; FPM-NOT: FunctionPass Manager
 
 define void @f() {
   ret void
 }