[LPM] Port CGProfilePass from NPM to LPM

[zequanwu]

update test.

[MaskRay]

llvm/lib/Transforms/Instrumentation/CGProfile.cpp

28

a static function does not need to be wrapped inside an unnamed namespace

100–105

wrap a struct/class in an unnamed namespace to prevent name collision

[hans]

Looks great, thanks! Just some minor comments.

llvm/include/llvm/InitializePasses.h

199

I think these declarations should be in alphabetic order.

llvm/include/llvm/Transforms/IPO.h

285

ultra nit: I would probably keep the empty line before the } here

llvm/lib/Transforms/Instrumentation/CGProfile.cpp

100–105

Might as well declare it 'final' while you're here.

[hans]

llvm/lib/Transforms/IPO/PassManagerBuilder.cpp

838

In the new pm this pass seems to be added conditionally based on some flag: if (PTO.CallGraphProfile)
Is it possible to do something similar here?

llvm/lib/Transforms/Instrumentation/CGProfile.cpp

129

Should it be "cg-profile" to match the npm pass name? Arthur has been working on making such pass names equal recently.

[zequanwu]

Address comments.

[zequanwu]

update.

[zequanwu]

Enable CGProfilePass for opt with LPM by default, like opt with NPM.

[MaskRay]

llvm/tools/opt/opt.cpp

281

If there is no strong need for tuning this, please delete the option and PassManagerBuilder::CallGraphProfile

---

I know that -enable-npm-call-graph-profile exists, but it seems like a temporary workaround for me. @zhizhouy @void (D62627) Is the option still used?

[zhizhouy]

llvm/tools/opt/opt.cpp

281

Does GNU assembler recognize .cgprofile section now?

I think we should keep this option as long as there is still usage of other than integrated assembler.

[MaskRay]

llvm/tools/opt/opt.cpp

281

Does GNU assembler recognize .cgprofile section now?

I don't think it will ever support this section.

I think we should keep this option as long as there is still usage of other than integrated assembler.

Can you give a link about the use case?

[zhizhouy]

llvm/tools/opt/opt.cpp

281

I have a link of the effort to migrate Linux kernel to integrated assembler in ChromeOS: https://bugs.chromium.org/p/chromium/issues/detail?id=1020923

I think they are only to migrate newer versions thus the old linux kernels are still using GNU assembler, and all kernels are built with LLVM now.

[MaskRay]

llvm/tools/opt/opt.cpp

281

If we have to have an option, we should make both legacy pm and new pm use the same option. We should not create two cl::opt.

[echristo]

Adding Chandler and Alina here as well.

In general, I don't think that this is such a great idea. Being able to have this sort of thing work more reliably is one of the reasons for the new pass manager. I think I'd like to see this split out into an old versus new pass manager pass to avoid the difficulty of cleaning this up after we finish migrating llvm to the new pass manager. This also seems to add some technical debt around options and other enablement which is also less than ideal. Is this compelling to add right now versus finishing work migrating llvm completely to the new pass manager and removing the old one? From speaking with Alina I think that work should be done in a short while.

Thanks.

-eric

[aeubanks]

In D83013#2132070, @echristo wrote:

Adding Chandler and Alina here as well.

In general, I don't think that this is such a great idea. Being able to have this sort of thing work more reliably is one of the reasons for the new pass manager. I think I'd like to see this split out into an old versus new pass manager pass to avoid the difficulty of cleaning this up after we finish migrating llvm to the new pass manager. This also seems to add some technical debt around options and other enablement which is also less than ideal. Is this compelling to add right now versus finishing work migrating llvm completely to the new pass manager and removing the old one? From speaking with Alina I think that work should be done in a short while.

Thanks.

-eric

I don't think we're that close yet, probably at least a couple months out, there are lots of loose ends to be tied up. I'll make a post soon in llvm-dev (maybe first we can sync up again) about what I think needs to be done before the NPM switch.

[hans]

In D83013#2132070, @echristo wrote:

Adding Chandler and Alina here as well.

In general, I don't think that this is such a great idea. Being able to have this sort of thing work more reliably is one of the reasons for the new pass manager. I think I'd like to see this split out into an old versus new pass manager pass to avoid the difficulty of cleaning this up after we finish migrating llvm to the new pass manager. This also seems to add some technical debt around options and other enablement which is also less than ideal. Is this compelling to add right now versus finishing work migrating llvm completely to the new pass manager and removing the old one? From speaking with Alina I think that work should be done in a short while.

Given how long the new pass manager has been in progress, we definitely don't want to block on enabling it. So yes, porting this pass to the current pass manager is compelling to do right now. I also don't see why it should be a big deal.

As for splitting it into separate passes, this patch technically does that, although it extracts and changes the core code a bit so it can be shared between the passes. I think that's how most passes have been adapted to work with both pass managers, no?

[zequanwu]

Delete enable-npm-call-graph-profile option for NPM, using enable-call-graph-profile for both LPM and NPM.

[asbirlea]

In D83013#2132088, @aeubanks wrote:

In D83013#2132070, @echristo wrote:

Adding Chandler and Alina here as well.

In general, I don't think that this is such a great idea. Being able to have this sort of thing work more reliably is one of the reasons for the new pass manager. I think I'd like to see this split out into an old versus new pass manager pass to avoid the difficulty of cleaning this up after we finish migrating llvm to the new pass manager. This also seems to add some technical debt around options and other enablement which is also less than ideal. Is this compelling to add right now versus finishing work migrating llvm completely to the new pass manager and removing the old one? From speaking with Alina I think that work should be done in a short while.

Thanks.

-eric

I don't think we're that close yet, probably at least a couple months out, there are lots of loose ends to be tied up. I'll make a post soon in llvm-dev (maybe first we can sync up again) about what I think needs to be done before the NPM switch.

+1 to sync up again and make progress towards the NPM switch.

I don't want to block this patch, but I do agree with Eric's point. We *really* want to focus more on the switch then invest into more LPM infra. Short term resolutions to unblock folks, with the best split possible, sure, keeping in mind they'll need to be cleaned up. But I don't want us to lose focus on tying up the remaining loose ends for the switch.
I think it's critical for LLVM's codebase health to focus on the NPM switch in the next couple of months.

[hans]

I don't want to block this patch, but I do agree with Eric's point. We *really* want to focus more on the switch then invest into more LPM infra. Short term resolutions to unblock folks, with the best split possible, sure, keeping in mind they'll need to be cleaned up.

Sounds good to me.

[hans]

LGTM

[nikic]

llvm/test/Other/opt-O2-pipeline.ll

289

Is it possible to switch this pass to use LazyBPI / LazyBFA, only fetched if PGO is actually in use?

PGO functionality that most people don't use adding expensive analysis passes like PDT should be avoided.

[hans]

llvm/lib/Transforms/IPO/PassManagerBuilder.cpp

170

Oh, just noticed: I think CallGraphProfile should be initialized along with the other flags here.

llvm/test/Other/opt-O2-pipeline.ll

289

I wonder if just switching to LazyBlockFrequencyInfo would help though. It looks to me like the CGProfile would request info about each function anyway.

I was surprised to see that Clang sets Opts.CallGraphProfile solely based on whether the integrated assembler is used. Maybe a better fix is to only set that to true when a profile is actually being used?

[nikic]

llvm/test/Other/opt-O2-pipeline.ll

289

I wonder if just switching to LazyBlockFrequencyInfo would help though. It looks to me like the CGProfile would request info about each function anyway.

It would only help if there is some way to only fetch the analysis conditionally. I believe many PGO passes use something like PSI.hasProfileSummary() or F.hasProfileData() for that.

I was surprised to see that Clang sets Opts.CallGraphProfile solely based on whether the integrated assembler is used. Maybe a better fix is to only set that to true when a profile is actually being used?

Right, just disabling this by default in clang/opt would also work.

For reference, the current compile-time numbers for this patch: https://llvm-compile-time-tracker.com/compare.php?from=516ff1d4baee28b1911737e47b42973567adf8ff&to=8df840660bb764b6653fcfd9ac7a72cc6adebde6&stat=instructions Not huge, but it adds up (some similar regressions have been introduced in LLVM 10).

[zequanwu]

llvm/test/Other/opt-O2-pipeline.ll

289

Do you mean disabling it just for LPM or both?

[zequanwu]

llvm/test/Other/opt-O2-pipeline.ll

289

I was surprised to see that Clang sets Opts.CallGraphProfile solely based on whether the integrated assembler is used. Maybe a better fix is to only set that to true when a profile is actually being used?

For Clang, a better fix I think is that Opts.CallGraphProfile should based on both whether the integrated assembler is used and whether profile instrumentation is turned on. What do you think?

[MaskRay]

llvm/test/Other/opt-O2-pipeline.ll

289

I'd prefer not having CallGraphProfile

• -no-integrated-as -S => no .cgprofile (.llvm_addrsig behaves this way)
• -S -> .cgprofile

[zequanwu]

llvm/test/Other/opt-O2-pipeline.ll

289

As discussed above, I think CGProfilePass should be disabled by default in clang unless -no-integrated-as is not given and -fprofile-instrument-use-path= is given. So, Opts.CallGraphProfile is a convenient switch for that.

[zequanwu]

• Disable enable-call-graph-profile by default in opt.
• Disable CGProfilePass by default in clang unless -no-integrated-as is not given and -fprofile-instrument-use-path= is given, as this pass only generates module metadata when profile data is given.

[MaskRay]

I still haven't seen a strong argument keeping a command line option -enable-npm-call-graph-profile. Asked in D62627.

Opts.getProfileUse() != CodeGenOptions::ProfileNone in

Opts.CallGraphProfile = Opts.getProfileUse() != CodeGenOptions::ProfileNone &&
                          !Opts.DisableIntegratedAS;

is redundant. CGProfile.cpp is a no-op if no function provides getEntryFreq().

[hans]

In D83013#2137607, @MaskRay wrote:

Opts.getProfileUse() != CodeGenOptions::ProfileNone in

Opts.CallGraphProfile = Opts.getProfileUse() != CodeGenOptions::ProfileNone &&
                          !Opts.DisableIntegratedAS;

is redundant. CGProfile.cpp is a no-op if no function provides getEntryFreq().

It's a functional no-op, but it runs the BFI analysis, which as Nikita pointed out above adds some compile-time cost. Not scheduling the pass unless we're using profile info seems like a reasonable way to avoid that cost to me.

The alternative of using LazyBlockFrequencyInfoPass and checking PSI->hasProfileSummary() first would also work I guess. If you think that's cleaner, maybe that's the better way to go.

[zequanwu]

• Remove "enable-call-graph-profile" option and enable CGProfilePass by default, unless -no-integrated-as is given in clang.
• Use LazyBlockFrequencyInfoPass instead of BlockFrequencyInfoWrapperPass and check F.getEntryCount before get BFI to reduce cost.

[zequanwu]

The alternative of using LazyBlockFrequencyInfoPass and checking PSI->hasProfileSummary() first would also work I guess. If you think that's cleaner, maybe that's the better way to go.

Since PSI->hasProfileSummary() is not necessary for this pass, it relies on function entry count. So, I check for F.getEntryCount() before getting BFI.

[MaskRay]

In D83013#2139882, @zequanwu wrote:

The alternative of using LazyBlockFrequencyInfoPass and checking PSI->hasProfileSummary() first would also work I guess. If you think that's cleaner, maybe that's the better way to go.

Since PSI->hasProfileSummary() is not necessary for this pass, it relies on function entry count. So, I check for F.getEntryCount() before getting BFI.

Thanks. The last update looks good to me. I'll defer the approval to @nikic and folks who have expressed concerns about deleting legacy PM.

[nikic]

LG from my side.

New compile-time numbers: https://llvm-compile-time-tracker.com/compare.php?from=0b39d2d75275b80994dac06b7ad05031cbd09393&to=fd070b79e063fff2fad3cd4a467f64dfca83eb90&stat=instructions It's nearly neutral now.

llvm/test/CodeGen/AMDGPU/opt-pipeline.ll

285

This test is out of date.

[hans]

In D83013#2142271, @nikic wrote:

New compile-time numbers: https://llvm-compile-time-tracker.com/compare.php?from=0b39d2d75275b80994dac06b7ad05031cbd09393&to=fd070b79e063fff2fad3cd4a467f64dfca83eb90&stat=instructions It's nearly neutral now.

Sounds great!

lgtm2 (with the test update Nikita mentioned)

[zequanwu]

Update test case.

[zequanwu]

rebase.

[echristo]

Some inline nits. I see you've already committed and that's fine - I still don't think we should do it, but we can delete it again soon :)

clang/lib/CodeGen/BackendUtil.cpp

623

Comment here as to why.

1148

Comment here as to why.

1566

Ditto :)

llvm/lib/Transforms/Instrumentation/CGProfile.cpp

63

Extra space? Did clang-format put this in?

65

Comment? What's the change for?

[lebedev.ri]

This seems to have broken the build
http://45.33.8.238/linux/22500/step_7.txt

[zequanwu]

Add comments and fix test failure in http://45.33.8.238/linux/22500/step_7.txt.

[zequanwu]

llvm/lib/Transforms/Instrumentation/CGProfile.cpp

63

Yes, clang-format put this in.

[hans]

Still lgtm. For what it's worth, I think you could have just re-committed with the fixes rather than uploading for review again.

[zequanwu]

In D83013#2143470, @hans wrote:

Still lgtm. For what it's worth, I think you could have just re-committed with the fixes rather than uploading for review again.

Gotcha, thanks.

[MaskRay]

In D83013#2143470, @hans wrote:

Still lgtm. For what it's worth, I think you could have just re-committed with the fixes rather than uploading for review again.

This may be a difference of habits but I usually upload the last revision if it contains anything more than comment changes. The reviewed version might be read by posterity to get a quick overview about the patch. Browsing git log -p is not very convenient at times.

clang/lib/CodeGen/BackendUtil.cpp

PMBuilder.Inliner = createFunctionInliningPass(
         (!CodeGenOpts.SampleProfileFile.empty() &&
          CodeGenOpts.PrepareForThinLTO));
   }
 
   PMBuilder.OptLevel = CodeGenOpts.OptimizationLevel;
   PMBuilder.SizeLevel = CodeGenOpts.OptimizeSize;
   PMBuilder.SLPVectorize = CodeGenOpts.VectorizeSLP;
   PMBuilder.LoopVectorize = CodeGenOpts.VectorizeLoop;
+  PMBuilder.CallGraphProfile = CodeGenOpts.CallGraphProfile;
 
   PMBuilder.DisableUnrollLoops = !CodeGenOpts.UnrollLoops;
   // Loop interleaving in the loop vectorizer has historically been set to be
   // enabled when loop unrolling is enabled.
   PMBuilder.LoopsInterleaved = CodeGenOpts.UnrollLoops;
   PMBuilder.MergeFunctions = CodeGenOpts.MergeFunctions;
   PMBuilder.PrepareForThinLTO = CodeGenOpts.PrepareForThinLTO;
   PMBuilder.PrepareForLTO = CodeGenOpts.PrepareForLTO;
void EmitAssemblyHelper::EmitAssemblyWithNewPassManager(
 
   PipelineTuningOptions PTO;
   PTO.LoopUnrolling = CodeGenOpts.UnrollLoops;
   // For historical reasons, loop interleaving is set to mirror setting for loop
   // unrolling.
   PTO.LoopInterleaving = CodeGenOpts.UnrollLoops;
   PTO.LoopVectorization = CodeGenOpts.VectorizeLoop;
   PTO.SLPVectorization = CodeGenOpts.VectorizeSLP;
   PTO.CallGraphProfile = CodeGenOpts.CallGraphProfile;
   PTO.Coroutines = LangOpts.Coroutines;
 
   PassInstrumentationCallbacks PIC;
   StandardInstrumentations SI;
   SI.registerCallbacks(PIC);
   PassBuilder PB(TM.get(), PTO, PGOOpt, &PIC);
 
   // Attempt to load pass plugins and register their callbacks with PB.
static void runThinLTOBackend(
   initTargetOptions(Diags, Conf.Options, CGOpts, TOpts, LOpts, HeaderOpts);
   Conf.SampleProfile = std::move(SampleProfile);
   Conf.PTO.LoopUnrolling = CGOpts.UnrollLoops;
   // For historical reasons, loop interleaving is set to mirror setting for loop
   // unrolling.
   Conf.PTO.LoopInterleaving = CGOpts.UnrollLoops;
   Conf.PTO.LoopVectorization = CGOpts.VectorizeLoop;
   Conf.PTO.SLPVectorization = CGOpts.VectorizeSLP;
   Conf.PTO.CallGraphProfile = CGOpts.CallGraphProfile;
 
   // Context sensitive profile.
   if (CGOpts.hasProfileCSIRInstr()) {
     Conf.RunCSIRInstr = true;
     Conf.CSIRProfile = std::move(CGOpts.InstrProfileOutput);
   } else if (CGOpts.hasProfileCSIRUse()) {
     Conf.RunCSIRInstr = false;
     Conf.CSIRProfile = std::move(CGOpts.ProfileInstrumentUsePath);

llvm/include/llvm/InitializePasses.h

 void initializeCFGPrinterLegacyPassPass(PassRegistry&);
 void initializeCFGSimplifyPassPass(PassRegistry&);
 void initializeCFGuardPass(PassRegistry&);
 void initializeCFGuardLongjmpPass(PassRegistry&);
 void initializeCFGViewerLegacyPassPass(PassRegistry&);
 void initializeCFIInstrInserterPass(PassRegistry&);
 void initializeCFLAndersAAWrapperPassPass(PassRegistry&);
 void initializeCFLSteensAAWrapperPassPass(PassRegistry&);
+void initializeCGProfileLegacyPassPass(PassRegistry &);
 void initializeCallGraphDOTPrinterPass(PassRegistry&);
 void initializeCallGraphPrinterLegacyPassPass(PassRegistry&);
 void initializeCallGraphViewerPass(PassRegistry&);
 void initializeCallGraphWrapperPassPass(PassRegistry&);
 void initializeCallSiteSplittingLegacyPassPass(PassRegistry&);
 void initializeCalledValuePropagationLegacyPassPass(PassRegistry &);
 void initializeCodeGenPreparePass(PassRegistry&);
 void initializeConstantHoistingLegacyPassPass(PassRegistry&);

 void initializeInferAddressSpacesPass(PassRegistry&);
 void initializeInferFunctionAttrsLegacyPassPass(PassRegistry&);
 void initializeInjectTLIMappingsLegacyPass(PassRegistry &);
 void initializeInlineCostAnalysisPass(PassRegistry&);
 void initializeInstCountPass(PassRegistry&);
 void initializeInstNamerPass(PassRegistry&);
 void initializeInstSimplifyLegacyPassPass(PassRegistry &);
 void initializeInstrProfilingLegacyPassPass(PassRegistry&);
 void initializeInstrOrderFileLegacyPassPass(PassRegistry&);
 void initializeInstructionCombiningPassPass(PassRegistry&);
 void initializeInstructionSelectPass(PassRegistry&);
 void initializeInterleavedAccessPass(PassRegistry&);
 void initializeInterleavedLoadCombinePass(PassRegistry &);
 void initializeInternalizeLegacyPassPass(PassRegistry&);
 void initializeIntervalPartitionPass(PassRegistry&);
 void initializeJumpThreadingPass(PassRegistry&);
 void initializeLCSSAVerificationPassPass(PassRegistry&);

llvm/include/llvm/Transforms/IPO.h

 // IR metadata to reflect the profile.
 ModulePass *createSampleProfileLoaderPass();
 ModulePass *createSampleProfileLoaderPass(StringRef Name);
 
 /// Write ThinLTO-ready bitcode to Str.
 ModulePass *createWriteThinLTOBitcodePass(raw_ostream &Str,
                                           raw_ostream *ThinLinkOS = nullptr);
 
+ModulePass *createCGProfileLegacyPass();
+
 } // End llvm namespace
 
 #endif

llvm/include/llvm/Transforms/IPO/PassManagerBuilder.h

public:
 
   /// The module summary index to use for importing information to the
   /// thin LTO backends, for example for the CFI and devirtualization type
   /// tests.
   const ModuleSummaryIndex *ImportSummary = nullptr;
 
   bool DisableTailCalls;
   bool DisableUnrollLoops;
+  bool CallGraphProfile;
   bool SLPVectorize;
   bool LoopVectorize;
   bool LoopsInterleaved;
   bool RerollLoops;
   bool NewGVN;
   bool DisableGVNLoadPRE;
   bool ForgetAllSCEVInLoopUnroll;
   bool VerifyInput;

llvm/include/llvm/Transforms/Instrumentation/CGProfile.h

 
 #include "llvm/ADT/MapVector.h"
 #include "llvm/IR/PassManager.h"
 
 namespace llvm {
 class CGProfilePass : public PassInfoMixin<CGProfilePass> {
 public:
   PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
-
-private:
-  void addModuleFlags(
-      Module &M,
-      MapVector<std::pair<Function *, Function *>, uint64_t> &Counts) const;
 };
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_CGPROFILE_H

llvm/lib/Transforms/IPO/PassManagerBuilder.cpp

cl::values(clEnumValN(AttributorRunOption::ALL, "all",
                           "enable module-wide attributor runs"),
                clEnumValN(AttributorRunOption::CGSCC, "cgscc",
                           "enable call graph SCC attributor runs"),
                clEnumValN(AttributorRunOption::NONE, "none",
                           "disable attributor runs")));
 
 extern cl::opt<bool> EnableKnowledgeRetention;
 
 PassManagerBuilder::PassManagerBuilder() {
     OptLevel = 2;
     SizeLevel = 0;
     LibraryInfo = nullptr;
     Inliner = nullptr;
     DisableUnrollLoops = false;
     SLPVectorize = false;
     LoopVectorize = true;
     LoopsInterleaved = true;
void PassManagerBuilder::populateModulePassManager(
   // See comment in the new PM for justification of scheduling splitting at
   // this stage (\ref buildModuleSimplificationPipeline).
   if (EnableHotColdSplit && !(PrepareForLTO || PrepareForThinLTO))
     MPM.add(createHotColdSplittingPass());
 
   if (MergeFunctions)
     MPM.add(createMergeFunctionsPass());
 
+  // Add Module flag "CG Profile" based on Branch Frequency Information.
+  if (CallGraphProfile)
+    MPM.add(createCGProfileLegacyPass());
+
   // LoopSink pass sinks instructions hoisted by LICM, which serves as a
   // canonicalization pass that enables other optimizations. As a result,
   // LoopSink pass needs to be a very late IR pass to avoid undoing LICM
   // result too early.
   MPM.add(createLoopSinkPass());
   // Get rid of LCSSA nodes.
   MPM.add(createInstSimplifyLegacyPass());
 

llvm/lib/Transforms/Instrumentation/CGProfile.cpp

 
 #include "llvm/ADT/MapVector.h"
 #include "llvm/Analysis/BlockFrequencyInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/MDBuilder.h"
 #include "llvm/IR/PassManager.h"
+#include "llvm/InitializePasses.h"
 #include "llvm/ProfileData/InstrProf.h"
+#include "llvm/Transforms/IPO.h"
 #include "llvm/Transforms/Instrumentation.h"
 
 #include <array>
 
 using namespace llvm;
 
-PreservedAnalyses CGProfilePass::run(Module &M, ModuleAnalysisManager &MAM) {
+static bool
+addModuleFlags(Module &M,
+               MapVector<std::pair<Function *, Function *>, uint64_t> &Counts) {
+  if (Counts.empty())
+    return false;
+
+  LLVMContext &Context = M.getContext();
+  MDBuilder MDB(Context);
+  std::vector<Metadata *> Nodes;
+
+  for (auto E : Counts) {
+    Metadata *Vals[] = {ValueAsMetadata::get(E.first.first),
+                        ValueAsMetadata::get(E.first.second),
+                        MDB.createConstant(ConstantInt::get(
+                            Type::getInt64Ty(Context), E.second))};
+    Nodes.push_back(MDNode::get(Context, Vals));
+  }
+
+  M.addModuleFlag(Module::Append, "CG Profile", MDNode::get(Context, Nodes));
+  return true;
+}
+
+static bool
+runCGProfilePass(Module &M,
+                 function_ref<BlockFrequencyInfo &(Function &)> GetBFI,
+                 function_ref<TargetTransformInfo &(Function &)> GetTTI) {
   MapVector<std::pair<Function *, Function *>, uint64_t> Counts;
-  FunctionAnalysisManager &FAM =
-      MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
   InstrProfSymtab Symtab;
   auto UpdateCounts = [&](TargetTransformInfo &TTI, Function *F,
                           Function *CalledF, uint64_t NewCount) {
     if (!CalledF || !TTI.isLoweredToCall(CalledF))
       return;
     uint64_t &Count = Counts[std::make_pair(F, CalledF)];
     Count = SaturatingAdd(Count, NewCount);
   };
   // Ignore error here.  Indirect calls are ignored if this fails.
-  (void)(bool)Symtab.create(M);
+  (void)(bool) Symtab.create(M);
   for (auto &F : M) {
     if (F.isDeclaration())
       continue;
-    auto &BFI = FAM.getResult<BlockFrequencyAnalysis>(F);
+    auto &BFI = GetBFI(F);
     if (BFI.getEntryFreq() == 0)
       continue;
-    TargetTransformInfo &TTI = FAM.getResult<TargetIRAnalysis>(F);
+    TargetTransformInfo &TTI = GetTTI(F);
     for (auto &BB : F) {
       Optional<uint64_t> BBCount = BFI.getBlockProfileCount(&BB);
       if (!BBCount)
         continue;
       for (auto &I : BB) {
         CallBase *CB = dyn_cast<CallBase>(&I);
         if (!CB)
           continue;
for (auto &BB : F) {
           }
           continue;
         }
         UpdateCounts(TTI, &F, CB->getCalledFunction(), *BBCount);
       }
     }
   }
 
-  addModuleFlags(M, Counts);
+  return addModuleFlags(M, Counts);
+}
 
-  return PreservedAnalyses::all();
+namespace {
+struct CGProfileLegacyPass final : public ModulePass {
+  static char ID;
+  CGProfileLegacyPass() : ModulePass(ID) {
+    initializeCGProfileLegacyPassPass(*PassRegistry::getPassRegistry());
   }
 
-void CGProfilePass::addModuleFlags(
-    Module &M,
-    MapVector<std::pair<Function *, Function *>, uint64_t> &Counts) const {
-  if (Counts.empty())
-    return;
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesCFG();
+    AU.addRequired<BlockFrequencyInfoWrapperPass>();
+    AU.addRequired<TargetTransformInfoWrapperPass>();
+  }
 
-  LLVMContext &Context = M.getContext();
-  MDBuilder MDB(Context);
-  std::vector<Metadata *> Nodes;
+  bool runOnModule(Module &M) override {
+    auto GetBFI = [this](Function &F) -> BlockFrequencyInfo & {
+      return this->getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI();
+    };
+    auto GetTTI = [this](Function &F) -> TargetTransformInfo & {
+      return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
+    };
 
-  for (auto E : Counts) {
-    Metadata *Vals[] = {ValueAsMetadata::get(E.first.first),
-                        ValueAsMetadata::get(E.first.second),
-                        MDB.createConstant(ConstantInt::get(
-                            Type::getInt64Ty(Context), E.second))};
-    Nodes.push_back(MDNode::get(Context, Vals));
+    return runCGProfilePass(M, GetBFI, GetTTI);
   }
+};
 
-  M.addModuleFlag(Module::Append, "CG Profile", MDNode::get(Context, Nodes));
+} // namespace
+
+char CGProfileLegacyPass::ID = 0;
+
+INITIALIZE_PASS(CGProfileLegacyPass, "cg-profile", "Call Graph Profile", false,
+                false)
+
+ModulePass *llvm::createCGProfileLegacyPass() {
+  return new CGProfileLegacyPass();
+}
+
+PreservedAnalyses CGProfilePass::run(Module &M, ModuleAnalysisManager &MAM) {
+  FunctionAnalysisManager &FAM =
+      MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
+  auto GetBFI = [&FAM](Function &F) -> BlockFrequencyInfo & {
+    return FAM.getResult<BlockFrequencyAnalysis>(F);
+  };
+  auto GetTTI = [&FAM](Function &F) -> TargetTransformInfo & {
+    return FAM.getResult<TargetIRAnalysis>(F);
+  };
+
+  runCGProfilePass(M, GetBFI, GetTTI);
+
+  return PreservedAnalyses::all();
 }

llvm/lib/Transforms/Instrumentation/Instrumentation.cpp

void llvm::initializeInstrumentation(PassRegistry &Registry) {
   initializeModuleAddressSanitizerLegacyPassPass(Registry);
   initializeBoundsCheckingLegacyPassPass(Registry);
   initializeControlHeightReductionLegacyPassPass(Registry);
   initializeGCOVProfilerLegacyPassPass(Registry);
   initializePGOInstrumentationGenLegacyPassPass(Registry);
   initializePGOInstrumentationUseLegacyPassPass(Registry);
   initializePGOIndirectCallPromotionLegacyPassPass(Registry);
   initializePGOMemOPSizeOptLegacyPassPass(Registry);
+  initializeCGProfileLegacyPassPass(Registry);
   initializeInstrOrderFileLegacyPassPass(Registry);
   initializeInstrProfilingLegacyPassPass(Registry);
   initializeMemorySanitizerLegacyPassPass(Registry);
   initializeHWAddressSanitizerLegacyPassPass(Registry);
   initializeThreadSanitizerLegacyPassPass(Registry);
   initializeModuleSanitizerCoverageLegacyPassPass(Registry);
   initializeDataFlowSanitizerPass(Registry);
 }
 
 /// LLVMInitializeInstrumentation - C binding for
 /// initializeInstrumentation.
 void LLVMInitializeInstrumentation(LLVMPassRegistryRef R) {
   initializeInstrumentation(*unwrap(R));
 }

llvm/test/CodeGen/AMDGPU/opt-pipeline.ll

 ; GCN-O1-NEXT:       Loop Pass Manager
 ; GCN-O1-NEXT:         Loop Invariant Code Motion
 ; GCN-O1-NEXT:       Lazy Branch Probability Analysis
 ; GCN-O1-NEXT:       Lazy Block Frequency Analysis
 ; GCN-O1-NEXT:       Optimization Remark Emitter
 ; GCN-O1-NEXT:       Warn about non-applied transformations
 ; GCN-O1-NEXT:       Alignment from assumptions
 ; GCN-O1-NEXT:     Strip Unused Function Prototypes
+; GCN-O1-NEXT:     Call Graph Profile
+; GCN-O1-NEXT:       FunctionPass Manager
+; GCN-O1-NEXT:         Dominator Tree Construction
+; GCN-O1-NEXT:         Natural Loop Information
+; GCN-O1-NEXT:         Post-Dominator Tree Construction
+; GCN-O1-NEXT:         Branch Probability Analysis
+; GCN-O1-NEXT:         Block Frequency Analysis
 ; GCN-O1-NEXT:     FunctionPass Manager
 ; GCN-O1-NEXT:       Dominator Tree Construction
 ; GCN-O1-NEXT:       Natural Loop Information
 ; GCN-O1-NEXT:       Post-Dominator Tree Construction
 ; GCN-O1-NEXT:       Branch Probability Analysis
 ; GCN-O1-NEXT:       Block Frequency Analysis
 ; GCN-O1-NEXT:       Canonicalize natural loops
 ; GCN-O1-NEXT:       LCSSA Verifier

 ; GCN-O2-NEXT:       Lazy Branch Probability Analysis
 ; GCN-O2-NEXT:       Lazy Block Frequency Analysis
 ; GCN-O2-NEXT:       Optimization Remark Emitter
 ; GCN-O2-NEXT:       Warn about non-applied transformations
 ; GCN-O2-NEXT:       Alignment from assumptions
 ; GCN-O2-NEXT:     Strip Unused Function Prototypes
 ; GCN-O2-NEXT:     Dead Global Elimination
 ; GCN-O2-NEXT:     Merge Duplicate Global Constants
+; GCN-O2-NEXT:     Call Graph Profile
+; GCN-O2-NEXT:       FunctionPass Manager
+; GCN-O2-NEXT:         Dominator Tree Construction
+; GCN-O2-NEXT:         Natural Loop Information
+; GCN-O2-NEXT:         Post-Dominator Tree Construction
+; GCN-O2-NEXT:         Branch Probability Analysis
+; GCN-O2-NEXT:         Block Frequency Analysis
 ; GCN-O2-NEXT:     FunctionPass Manager
 ; GCN-O2-NEXT:       Dominator Tree Construction
 ; GCN-O2-NEXT:       Natural Loop Information
 ; GCN-O2-NEXT:       Post-Dominator Tree Construction
 ; GCN-O2-NEXT:       Branch Probability Analysis
 ; GCN-O2-NEXT:       Block Frequency Analysis
 ; GCN-O2-NEXT:       Canonicalize natural loops
 ; GCN-O2-NEXT:       LCSSA Verifier

 ; GCN-O3-NEXT:       Lazy Branch Probability Analysis
 ; GCN-O3-NEXT:       Lazy Block Frequency Analysis
 ; GCN-O3-NEXT:       Optimization Remark Emitter
 ; GCN-O3-NEXT:       Warn about non-applied transformations
 ; GCN-O3-NEXT:       Alignment from assumptions
 ; GCN-O3-NEXT:     Strip Unused Function Prototypes
 ; GCN-O3-NEXT:     Dead Global Elimination
 ; GCN-O3-NEXT:     Merge Duplicate Global Constants
+; GCN-O3-NEXT:     Call Graph Profile
+; GCN-O3-NEXT:       FunctionPass Manager
+; GCN-O3-NEXT:         Dominator Tree Construction
+; GCN-O3-NEXT:         Natural Loop Information
+; GCN-O3-NEXT:         Post-Dominator Tree Construction
+; GCN-O3-NEXT:         Branch Probability Analysis
+; GCN-O3-NEXT:         Block Frequency Analysis
 ; GCN-O3-NEXT:     FunctionPass Manager
 ; GCN-O3-NEXT:       Dominator Tree Construction
 ; GCN-O3-NEXT:       Natural Loop Information
 ; GCN-O3-NEXT:       Post-Dominator Tree Construction
 ; GCN-O3-NEXT:       Branch Probability Analysis
 ; GCN-O3-NEXT:       Block Frequency Analysis
 ; GCN-O3-NEXT:       Canonicalize natural loops
 ; GCN-O3-NEXT:       LCSSA Verifier

llvm/test/Instrumentation/cgprofile.ll

 ; RUN: opt < %s -passes cg-profile -S | FileCheck %s
+; RUN: opt < %s -cg-profile -S | FileCheck %s
 
 declare void @b()
 
 define void @a() !prof !1 {
   call void @b()
   ret void
 }
 

llvm/test/Other/opt-O2-pipeline.ll

 ; CHECK-NEXT:       Lazy Branch Probability Analysis
 ; CHECK-NEXT:       Lazy Block Frequency Analysis
 ; CHECK-NEXT:       Optimization Remark Emitter
 ; CHECK-NEXT:       Warn about non-applied transformations
 ; CHECK-NEXT:       Alignment from assumptions
 ; CHECK-NEXT:     Strip Unused Function Prototypes
 ; CHECK-NEXT:     Dead Global Elimination
 ; CHECK-NEXT:     Merge Duplicate Global Constants
+; CHECK-NEXT:     Call Graph Profile
+; CHECK-NEXT:       FunctionPass Manager
+; CHECK-NEXT:         Dominator Tree Construction
+; CHECK-NEXT:         Natural Loop Information
+; CHECK-NEXT:         Post-Dominator Tree Construction
+; CHECK-NEXT:         Branch Probability Analysis
+; CHECK-NEXT:         Block Frequency Analysis
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Post-Dominator Tree Construction
 ; CHECK-NEXT:       Branch Probability Analysis
 ; CHECK-NEXT:       Block Frequency Analysis
 ; CHECK-NEXT:       Canonicalize natural loops
 ; CHECK-NEXT:       LCSSA Verifier

llvm/test/Other/opt-O3-pipeline.ll

 ; CHECK-NEXT:       Lazy Branch Probability Analysis
 ; CHECK-NEXT:       Lazy Block Frequency Analysis
 ; CHECK-NEXT:       Optimization Remark Emitter
 ; CHECK-NEXT:       Warn about non-applied transformations
 ; CHECK-NEXT:       Alignment from assumptions
 ; CHECK-NEXT:     Strip Unused Function Prototypes
 ; CHECK-NEXT:     Dead Global Elimination
 ; CHECK-NEXT:     Merge Duplicate Global Constants
+; CHECK-NEXT:     Call Graph Profile
+; CHECK-NEXT:       FunctionPass Manager
+; CHECK-NEXT:         Dominator Tree Construction
+; CHECK-NEXT:         Natural Loop Information
+; CHECK-NEXT:         Post-Dominator Tree Construction
+; CHECK-NEXT:         Branch Probability Analysis
+; CHECK-NEXT:         Block Frequency Analysis
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Post-Dominator Tree Construction
 ; CHECK-NEXT:       Branch Probability Analysis
 ; CHECK-NEXT:       Block Frequency Analysis
 ; CHECK-NEXT:       Canonicalize natural loops
 ; CHECK-NEXT:       LCSSA Verifier

llvm/test/Other/opt-Os-pipeline.ll

 ; CHECK-NEXT:       Lazy Branch Probability Analysis
 ; CHECK-NEXT:       Lazy Block Frequency Analysis
 ; CHECK-NEXT:       Optimization Remark Emitter
 ; CHECK-NEXT:       Warn about non-applied transformations
 ; CHECK-NEXT:       Alignment from assumptions
 ; CHECK-NEXT:     Strip Unused Function Prototypes
 ; CHECK-NEXT:     Dead Global Elimination
 ; CHECK-NEXT:     Merge Duplicate Global Constants
+; CHECK-NEXT:     Call Graph Profile
+; CHECK-NEXT:       FunctionPass Manager
+; CHECK-NEXT:         Dominator Tree Construction
+; CHECK-NEXT:         Natural Loop Information
+; CHECK-NEXT:         Post-Dominator Tree Construction
+; CHECK-NEXT:         Branch Probability Analysis
+; CHECK-NEXT:         Block Frequency Analysis
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Post-Dominator Tree Construction
 ; CHECK-NEXT:       Branch Probability Analysis
 ; CHECK-NEXT:       Block Frequency Analysis
 ; CHECK-NEXT:       Canonicalize natural loops
 ; CHECK-NEXT:       LCSSA Verifier

llvm/tools/opt/opt.cpp

static cl::opt<unsigned> TimeTraceGranularity(
     cl::desc("Minimum time granularity (in microseconds) traced by time profiler"),
     cl::init(500), cl::Hidden);
 
 static cl::opt<std::string>
     TimeTraceFile("time-trace-file",
                     cl::desc("Specify time trace file destination"),
                     cl::value_desc("filename"));
 
+static cl::opt<bool> EnableCallGraphProfile(
+    "enable-call-graph-profile", cl::init(true), cl::Hidden,
+    cl::desc("Enable call graph profile pass (default = on)"));
+
 static cl::opt<bool> RemarksWithHotness(
     "pass-remarks-with-hotness",
     cl::desc("With PGO, include profile count in optimization remarks"),
     cl::Hidden);
 
 static cl::opt<unsigned>
     RemarksHotnessThreshold("pass-remarks-hotness-threshold",
                             cl::desc("Minimum profile count required for "
static void AddOptimizationPasses(legacy::PassManagerBase &MPM,
   }
   Builder.DisableUnrollLoops = (DisableLoopUnrolling.getNumOccurrences() > 0) ?
                                DisableLoopUnrolling : OptLevel == 0;
 
   Builder.LoopVectorize = OptLevel > 1 && SizeLevel < 2;
 
   Builder.SLPVectorize = OptLevel > 1 && SizeLevel < 2;
 
+  Builder.CallGraphProfile = EnableCallGraphProfile;
+
   if (TM)
     TM->adjustPassManager(Builder);
 
   if (Coroutines)
     addCoroutinePassesToExtensionPoints(Builder);
 
   switch (PGOKindFlag) {
   case InstrGen: