Use CountingFunctionInserter both mcount and cygprofile calls, before and after inlining

[hans]

Summary

Clang implements the -finstrument-functions flag inherited from GCC, which inserts calls to __cyg_profile_func_{enter,exit} on function entry and exit.

This is useful for getting a trace of how the functions in a program are executed. Normally, the calls remain even if a function is inlined into another function, but it is useful to be able to turn this off for users who are interested in a lower-level trace, i.e. one that reflects what functions are called post-inlining. (We use this to generate link order files for Chromium.)

LLVM already has a pass for inserting similar instrumentation calls to mcount(), which it does after inlining. This patch renames and extends that pass to handle calls both to mcount and the cygprofile functions, before and/or after inlining as controlled by function attributes.

[hfinkel]

If you only want these calls post-inlining (and otherwise wish to minimize the effect on the rest of the optimizer), I'd recommend a different approach. What we did for mcount, which has similar requirements, was that we had the frontend add an attribute, and then a late pass actually added the function calls (lib/CodeGen/CountingFunctionInserter.cpp). I recommend extending CountingFunctionInserter to enable it to add these functions (based on some additional attribute), and then just add whatever attribute is necessary in the frontend based on some new flag.

[rnk]

In D39287#906565, @hfinkel wrote:

If you only want these calls post-inlining (and otherwise wish to minimize the effect on the rest of the optimizer), I'd recommend a different approach. What we did for mcount, which has similar requirements, was that we had the frontend add an attribute, and then a late pass actually added the function calls (lib/CodeGen/CountingFunctionInserter.cpp). I recommend extending CountingFunctionInserter to enable it to add these functions (based on some additional attribute), and then just add whatever attribute is necessary in the frontend based on some new flag.

+1. I know it's a big departure from how the existing feature is implemented, but it's consistent with every other major form of instrumentation that clang emits these days.

[hans]

In D39287#906565, @hfinkel wrote:

If you only want these calls post-inlining (and otherwise wish to minimize the effect on the rest of the optimizer),

As I mentioned in the description, the calls should generally be inserted before inlining and preserved. Doing them post-inlining should be an optional mode that I'm trying to add. This is sort of the crux of the problem, otherwise inserting the instrumentation later would be the obvious way to go.

I'd recommend a different approach. What we did for mcount, which has similar requirements, was that we had the frontend add an attribute, and then a late pass actually added the function calls (lib/CodeGen/CountingFunctionInserter.cpp). I recommend extending CountingFunctionInserter to enable it to add these functions (based on some additional attribute), and then just add whatever attribute is necessary in the frontend based on some new flag.

Would it be OK to run CountingFunctionInserter both before and after inlining and inserting these calls depending on some flag?

[hfinkel]

In D39287#907578, @hans wrote:

In D39287#906565, @hfinkel wrote:

If you only want these calls post-inlining (and otherwise wish to minimize the effect on the rest of the optimizer),

As I mentioned in the description, the calls should generally be inserted before inlining and preserved. Doing them post-inlining should be an optional mode that I'm trying to add. This is sort of the crux of the problem, otherwise inserting the instrumentation later would be the obvious way to go.

I understood. I figured that you would add them later in some mode. Otherwise, you would leave the logic as-is. As you suggest, you could have CountingFunctionInserter run both pre and post inlining and then not have duplicate logic.

I'd recommend a different approach. What we did for mcount, which has similar requirements, was that we had the frontend add an attribute, and then a late pass actually added the function calls (lib/CodeGen/CountingFunctionInserter.cpp). I recommend extending CountingFunctionInserter to enable it to add these functions (based on some additional attribute), and then just add whatever attribute is necessary in the frontend based on some new flag.

Would it be OK to run CountingFunctionInserter both before and after inlining and inserting these calls depending on some flag?

I'm not sure the logic of inserting the calls is complex enough to worry about reducing the duplication of having both logic in the frontend and in CountingFunctionInserter, but if you feel that it is, then I think this makes sense.

[hans]

Here's a new approach which renames and extends the existing CountingFunctionInserter to run both before and after inlining (with the old PM it's two passes actually) and handle both cygprofile and mcount calls as controlled by function attributes.

I haven't hooked up the Clang side yet, but I'd be interested to hear your thoughts on this approach.

[rnk]

I think this is the way to go. Thanks! :)

lib/Transforms/Utils/EntryExitInstrumenter.cpp

22–24

nit: mark these static, sink the anonymous namespace to the two pass structs.

57

nit: static

[hans]

Rebasing and addressing comments.

I've also added the pass in PassManagerBuilder::populateFunctionPassManager() because I realized that's how Clang sets up the passes.

One thing that worries me is that this set-up is pretty fragile against how the passes are run. I think it will currently work, but if the pass were to somehow run more than once, the instrumentation could get inserted multiple times. One way to avoid that would be to remove the attribute (or probably faster, give it an empty value?) when the pass "consumes" the attribute. Does that sound reasonable?

I also looked into consolidating this with the FEntryInserter pass which inserts calls to fentry. That turned out to be tricky however, because the calls are supposed to be inserted before the function prologue, which means the pass operates on MachineFunctions and inserts MachineInstrs. So I've left that out.

I've updated the Clang-side patch here: https://reviews.llvm.org/D39331

[rnk]

In D39287#922534, @hans wrote:

Rebasing and addressing comments.

I've also added the pass in PassManagerBuilder::populateFunctionPassManager() because I realized that's how Clang sets up the passes.

One thing that worries me is that this set-up is pretty fragile against how the passes are run. I think it will currently work, but if the pass were to somehow run more than once, the instrumentation could get inserted multiple times. One way to avoid that would be to remove the attribute (or probably faster, give it an empty value?) when the pass "consumes" the attribute. Does that sound reasonable?

What do other instrumentation passes do to defend against this? I guess they just don't have this problem because they don't appear twice in the pipeline?

I'm in favor of removing the function attribute when we instrument. AttributeLists are immutable, so nuking the whole attribute should be just as fast as setting the value.

I also looked into consolidating this with the FEntryInserter pass which inserts calls to fentry. That turned out to be tricky however, because the calls are supposed to be inserted before the function prologue, which means the pass operates on MachineFunctions and inserts MachineInstrs. So I've left that out.

Oh, wow, that's pretty special functionality. Seems best to leave it alone.

I've updated the Clang-side patch here: https://reviews.llvm.org/D39331

I think this is pretty uncontroversial, so let's remove the attribute on instrumentation, land this, and we can address any feedback from Hal or others in follow-ups.

[hans]

In D39287#923553, @rnk wrote:

In D39287#922534, @hans wrote:

Rebasing and addressing comments.

I've also added the pass in PassManagerBuilder::populateFunctionPassManager() because I realized that's how Clang sets up the passes.

One thing that worries me is that this set-up is pretty fragile against how the passes are run. I think it will currently work, but if the pass were to somehow run more than once, the instrumentation could get inserted multiple times. One way to avoid that would be to remove the attribute (or probably faster, give it an empty value?) when the pass "consumes" the attribute. Does that sound reasonable?

What do other instrumentation passes do to defend against this? I guess they just don't have this problem because they don't appear twice in the pipeline?

CountingFunctionInserter wouldn't appear in the pipeline twice becaues it runs so late, as part of the codegen passes (which is also why it lived under Codegen/). The sanitizer instrumentation passes avoid the problem by not being in the pipeline by default, but requiring e.g. Clang to insert them.

I'm in favor of removing the function attribute when we instrument. AttributeLists are immutable, so nuking the whole attribute should be just as fast as setting the value.

Done.

I've updated the Clang-side patch here: https://reviews.llvm.org/D39331

I think this is pretty uncontroversial, so let's remove the attribute on instrumentation, land this, and we can address any feedback from Hal or others in follow-ups.

Great, thanks. (I'll just wait until the Clang side is completely worked out)

include/llvm/CodeGen/Passes.h

namespace llvm {
   /// createUnreachableBlockEliminationPass - The LLVM code generator does not
   /// work well with unreachable basic blocks (what live ranges make sense for a
   /// block that cannot be reached?).  As such, a code generator should either
   /// not instruction select unreachable blocks, or run this pass as its
   /// last LLVM modifying pass to clean up blocks that are not reachable from
   /// the entry block.
   FunctionPass *createUnreachableBlockEliminationPass();
 
-  /// Insert mcount-like function calls.
-  FunctionPass *createCountingFunctionInserterPass();
-
   /// MachineFunctionPrinter pass - This pass prints out the machine function to
   /// the given stream as a debugging tool.
   MachineFunctionPass *
   createMachineFunctionPrinterPass(raw_ostream &OS,
                                    const std::string &Banner ="");
 
   /// MIRPrinting pass - this pass prints out the LLVM IR into the given stream
   /// using the MIR serialization format.

include/llvm/InitializePasses.h

 void initializeCallGraphWrapperPassPass(PassRegistry&);
 void initializeCodeGenPreparePass(PassRegistry&);
 void initializeConstantHoistingLegacyPassPass(PassRegistry&);
 void initializeCalledValuePropagationLegacyPassPass(PassRegistry &);
 void initializeConstantMergeLegacyPassPass(PassRegistry&);
 void initializeConstantPropagationPass(PassRegistry&);
 void initializeCorrelatedValuePropagationPass(PassRegistry&);
 void initializeCostModelAnalysisPass(PassRegistry&);
-void initializeCountingFunctionInserterPass(PassRegistry&);
+void initializeEntryExitInstrumenterPass(PassRegistry&);
+void initializePostInlineEntryExitInstrumenterPass(PassRegistry&);
 void initializeCrossDSOCFIPass(PassRegistry&);
 void initializeDAEPass(PassRegistry&);
 void initializeDAHPass(PassRegistry&);
 void initializeDCELegacyPassPass(PassRegistry&);
 void initializeDSELegacyPassPass(PassRegistry&);
 void initializeDataFlowSanitizerPass(PassRegistry&);
 void initializeDeadInstEliminationPass(PassRegistry&);
 void initializeDeadMachineInstructionElimPass(PassRegistry&);

include/llvm/LinkAllPasses.h

ForcePassLinking() {
       (void) llvm::createStripDeadDebugInfoPass();
       (void) llvm::createStripDeadPrototypesPass();
       (void) llvm::createTailCallEliminationPass();
       (void) llvm::createJumpThreadingPass();
       (void) llvm::createUnifyFunctionExitNodesPass();
       (void) llvm::createInstCountPass();
       (void) llvm::createConstantHoistingPass();
       (void) llvm::createCodeGenPreparePass();
-      (void) llvm::createCountingFunctionInserterPass();
+      (void) llvm::createEntryExitInstrumenterPass();
+      (void) llvm::createPostInlineEntryExitInstrumenterPass();
       (void) llvm::createEarlyCSEPass();
       (void) llvm::createGVNHoistPass();
       (void) llvm::createMergedLoadStoreMotionPass();
       (void) llvm::createGVNPass();
       (void) llvm::createNewGVNPass();
       (void) llvm::createMemCpyOptPass();
       (void) llvm::createLoopDeletionPass();
       (void) llvm::createPostDomTree();

include/llvm/Transforms/Scalar.h

 ModulePass *createNameAnonGlobalPass();
 
 //===----------------------------------------------------------------------===//
 //
 // LibCallsShrinkWrap - Shrink-wraps a call to function if the result is not
 // used.
 //
 FunctionPass *createLibCallsShrinkWrapPass();
+
+//===----------------------------------------------------------------------===//
+//
+// EntryExitInstrumenter pass - Instrument function entry/exit with calls to
+// mcount(), @__cyg_profile_func_{enter,exit} and the like. There are two
+// variants, intended to run pre- and post-inlining, respectively.
+//
+FunctionPass *createEntryExitInstrumenterPass();
+FunctionPass *createPostInlineEntryExitInstrumenterPass();
+
 } // End llvm namespace
 
 #endif

include/llvm/Transforms/Utils/EntryExitInstrumenter.h

+//===- EntryExitInstrumenter.h - Function Entry/Exit Instrumentation ------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// EntryExitInstrumenter pass - Instrument function entry/exit with calls to
+// mcount(), @__cyg_profile_func_{enter,exit} and the like. There are two
+// variants, intended to run pre- and post-inlining, respectively.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TRANSFORMS_UTILS_ENTRYEXITINSTRUMENTER_H
+#define LLVM_TRANSFORMS_UTILS_ENTRYEXITINSTRUMENTER_H
+
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+class Function;
+
+struct EntryExitInstrumenterPass
+    : public PassInfoMixin<EntryExitInstrumenterPass> {
+  EntryExitInstrumenterPass(bool PostInlining) : PostInlining(PostInlining) {}
+
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
+
+  bool PostInlining;
+};
+
+} // namespace llvm
+
+#endif // LLVM_TRANSFORMS_UTILS_ENTRYEXITINSTRUMENTER_H

lib/CodeGen/CMakeLists.txt

 add_llvm_library(LLVMCodeGen
   AggressiveAntiDepBreaker.cpp
   AllocationOrder.cpp
   Analysis.cpp
   AtomicExpandPass.cpp
   BasicTargetTransformInfo.cpp
   BranchFolding.cpp
   BranchRelaxation.cpp
   BuiltinGCs.cpp
   CalcSpillWeights.cpp
   CallingConvLower.cpp
   CodeGen.cpp
   CodeGenPrepare.cpp
-  CountingFunctionInserter.cpp
   CriticalAntiDepBreaker.cpp
   DeadMachineInstructionElim.cpp
   DetectDeadLanes.cpp
   DFAPacketizer.cpp
   DwarfEHPrepare.cpp
   EarlyIfConversion.cpp
   EdgeBundles.cpp
   ExecutionDepsFix.cpp

lib/CodeGen/CodeGen.cpp

 using namespace llvm;
 
 /// initializeCodeGen - Initialize all passes linked into the CodeGen library.
 void llvm::initializeCodeGen(PassRegistry &Registry) {
   initializeAtomicExpandPass(Registry);
   initializeBranchFolderPassPass(Registry);
   initializeBranchRelaxationPass(Registry);
   initializeCodeGenPreparePass(Registry);
-  initializeCountingFunctionInserterPass(Registry);
   initializeDeadMachineInstructionElimPass(Registry);
   initializeDetectDeadLanesPass(Registry);
   initializeDwarfEHPreparePass(Registry);
   initializeEarlyIfConverterPass(Registry);
   initializeExpandISelPseudosPass(Registry);
   initializeExpandMemCmpPassPass(Registry);
   initializeExpandPostRAPass(Registry);
   initializeFEntryInserterPass(Registry);

lib/CodeGen/CountingFunctionInserter.cpp

-//===- CountingFunctionInserter.cpp - Insert mcount-like function calls ---===//
-//
-//                     The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-//
-// Insert calls to counter functions, such as mcount, intended to be called
-// once per function, at the beginning of each function.
-//
-//===----------------------------------------------------------------------===//
-
-#include "llvm/Analysis/GlobalsModRef.h"
-#include "llvm/CodeGen/Passes.h"
-#include "llvm/IR/Function.h"
-#include "llvm/IR/Instructions.h"
-#include "llvm/IR/Module.h"
-#include "llvm/IR/Type.h"
-#include "llvm/Pass.h"
-using namespace llvm;
-
-namespace {
-  struct CountingFunctionInserter : public FunctionPass {
-    static char ID; // Pass identification, replacement for typeid
-    CountingFunctionInserter() : FunctionPass(ID) {
-      initializeCountingFunctionInserterPass(*PassRegistry::getPassRegistry());
-    }
-
-    void getAnalysisUsage(AnalysisUsage &AU) const override {
-      AU.addPreserved<GlobalsAAWrapperPass>();
-    }
-
-    bool runOnFunction(Function &F) override {
-      StringRef CountingFunctionName =
-        F.getFnAttribute("counting-function").getValueAsString();
-      if (CountingFunctionName.empty())
-        return false;
-
-      Type *VoidTy = Type::getVoidTy(F.getContext());
-      Constant *CountingFn =
-        F.getParent()->getOrInsertFunction(CountingFunctionName,
-                                           VoidTy);
-      CallInst::Create(CountingFn, "", &*F.begin()->getFirstInsertionPt());
-      return true;
-    }
-  };
-
-  char CountingFunctionInserter::ID = 0;
-}
-
-INITIALIZE_PASS(CountingFunctionInserter, "cfinserter",
-                "Inserts calls to mcount-like functions", false, false)
-
-FunctionPass *llvm::createCountingFunctionInserterPass() {
-  return new CountingFunctionInserter();
-}

lib/CodeGen/TargetPassConfig.cpp

void TargetPassConfig::addIRPasses() {
 
   // Prepare expensive constants for SelectionDAG.
   if (getOptLevel() != CodeGenOpt::None && !DisableConstantHoisting)
     addPass(createConstantHoistingPass());
 
   if (getOptLevel() != CodeGenOpt::None && !DisablePartialLibcallInlining)
     addPass(createPartiallyInlineLibCallsPass());
 
-  // Insert calls to mcount-like functions.
-  addPass(createCountingFunctionInserterPass());
+  // Instrument function entry and exit, e.g. with calls to mcount().
+  addPass(createPostInlineEntryExitInstrumenterPass());
 
   // Add scalarization of target's unsupported masked memory intrinsics pass.
   // the unsupported intrinsic will be replaced with a chain of basic blocks,
   // that stores/loads element one-by-one if the appropriate mask bit is set.
   addPass(createScalarizeMaskedMemIntrinPass());
 
   // Expand reduction intrinsics into shuffle sequences if the target wants to.
   addPass(createExpandReductionsPass());

lib/Passes/PassBuilder.cpp

 #include "llvm/Transforms/Scalar/SROA.h"
 #include "llvm/Transforms/Scalar/SimpleLoopUnswitch.h"
 #include "llvm/Transforms/Scalar/SimplifyCFG.h"
 #include "llvm/Transforms/Scalar/Sink.h"
 #include "llvm/Transforms/Scalar/SpeculativeExecution.h"
 #include "llvm/Transforms/Scalar/TailRecursionElimination.h"
 #include "llvm/Transforms/Utils/AddDiscriminators.h"
 #include "llvm/Transforms/Utils/BreakCriticalEdges.h"
+#include "llvm/Transforms/Utils/EntryExitInstrumenter.h"
 #include "llvm/Transforms/Utils/LCSSA.h"
 #include "llvm/Transforms/Utils/LibCallsShrinkWrap.h"
 #include "llvm/Transforms/Utils/LoopSimplify.h"
 #include "llvm/Transforms/Utils/LowerInvoke.h"
 #include "llvm/Transforms/Utils/Mem2Reg.h"
 #include "llvm/Transforms/Utils/NameAnonGlobals.h"
 #include "llvm/Transforms/Utils/PredicateInfo.h"
 #include "llvm/Transforms/Utils/SimplifyInstructions.h"

lib/Passes/PassRegistry.def

 FUNCTION_PASS("correlated-propagation", CorrelatedValuePropagationPass())
 FUNCTION_PASS("dce", DCEPass())
 FUNCTION_PASS("div-rem-pairs", DivRemPairsPass())
 FUNCTION_PASS("dse", DSEPass())
 FUNCTION_PASS("dot-cfg", CFGPrinterPass())
 FUNCTION_PASS("dot-cfg-only", CFGOnlyPrinterPass())
 FUNCTION_PASS("early-cse", EarlyCSEPass(/*UseMemorySSA=*/false))
 FUNCTION_PASS("early-cse-memssa", EarlyCSEPass(/*UseMemorySSA=*/true))
+FUNCTION_PASS("ee-instrument", EntryExitInstrumenterPass(/*PostInlining=*/false))
+FUNCTION_PASS("post-inline-ee-instrument", EntryExitInstrumenterPass(/*PostInlining=*/true))
 FUNCTION_PASS("gvn-hoist", GVNHoistPass())
 FUNCTION_PASS("instcombine", InstCombinePass())
 FUNCTION_PASS("instsimplify", InstSimplifierPass())
 FUNCTION_PASS("invalidate<all>", InvalidateAllAnalysesPass())
 FUNCTION_PASS("float2int", Float2IntPass())
 FUNCTION_PASS("no-op-function", NoOpFunctionPass())
 FUNCTION_PASS("libcalls-shrinkwrap", LibCallsShrinkWrapPass())
 FUNCTION_PASS("loweratomic", LowerAtomicPass())

lib/Transforms/IPO/PassManagerBuilder.cpp

void PassManagerBuilder::addInstructionCombiningPass(
     legacy::PassManagerBase &PM) const {
   bool ExpensiveCombines = OptLevel > 2;
   PM.add(createInstructionCombiningPass(ExpensiveCombines));
 }
 
 void PassManagerBuilder::populateFunctionPassManager(
     legacy::FunctionPassManager &FPM) {
   addExtensionsToPM(EP_EarlyAsPossible, FPM);
+  FPM.add(createEntryExitInstrumenterPass());
 
   // Add LibraryInfo if we have some.
   if (LibraryInfo)
     FPM.add(new TargetLibraryInfoWrapperPass(*LibraryInfo));
 
   if (OptLevel == 0) return;
 
   addInitialAliasAnalysisPasses(FPM);

lib/Transforms/Scalar/Scalar.cpp

void llvm::initializeScalarOpts(PassRegistry &Registry) {
   initializeStraightLineStrengthReducePass(Registry);
   initializePlaceBackedgeSafepointsImplPass(Registry);
   initializePlaceSafepointsPass(Registry);
   initializeFloat2IntLegacyPassPass(Registry);
   initializeLoopDistributeLegacyPass(Registry);
   initializeLoopLoadEliminationPass(Registry);
   initializeLoopSimplifyCFGLegacyPassPass(Registry);
   initializeLoopVersioningPassPass(Registry);
+  initializeEntryExitInstrumenterPass(Registry);
+  initializePostInlineEntryExitInstrumenterPass(Registry);
 }
 
 void LLVMInitializeScalarOpts(LLVMPassRegistryRef R) {
   initializeScalarOpts(*unwrap(R));
 }
 
 void LLVMAddAggressiveDCEPass(LLVMPassManagerRef PM) {
   unwrap(PM)->add(createAggressiveDCEPass());

lib/Transforms/Utils/CMakeLists.txt

 add_llvm_library(LLVMTransformUtils
   ASanStackFrameLayout.cpp
   AddDiscriminators.cpp
   BasicBlockUtils.cpp
   BreakCriticalEdges.cpp
   BuildLibCalls.cpp
   BypassSlowDivision.cpp
   CloneFunction.cpp
   CloneModule.cpp
   CodeExtractor.cpp
   CtorUtils.cpp
   DemoteRegToStack.cpp
+  EntryExitInstrumenter.cpp
   EscapeEnumerator.cpp
   Evaluator.cpp
   FlattenCFG.cpp
   FunctionComparator.cpp
   FunctionImportUtils.cpp
   GlobalStatus.cpp
   InlineFunction.cpp
   ImportedFunctionsInliningStatistics.cpp

lib/Transforms/Utils/EntryExitInstrumenter.cpp

+//===- EntryExitInstrumenter.cpp - Function Entry/Exit Instrumentation ----===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Transforms/Utils/EntryExitInstrumenter.h"
+#include "llvm/Analysis/GlobalsModRef.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/Type.h"
+#include "llvm/Pass.h"
+#include "llvm/Transforms/Scalar.h"
+using namespace llvm;
+
+static void insertCall(Function &CurFn, StringRef Func,
+                       Instruction *InsertionPt) {
+  Module &M = *InsertionPt->getParent()->getParent()->getParent();
+  LLVMContext &C = InsertionPt->getParent()->getContext();
+
+  if (Func == "mcount" ||
+      Func == ".mcount" ||
+      Func == "\01__gnu_mcount_nc" ||
+      Func == "\01_mcount" ||
+      Func == "\01mcount" ||
+      Func == "__mcount" ||
+      Func == "_mcount") {
+    Constant *Fn = M.getOrInsertFunction(Func, Type::getVoidTy(C));
+    CallInst::Create(Fn, "", InsertionPt);
+    return;
+  }
+
+  if (Func == "__cyg_profile_func_enter" || Func == "__cyg_profile_func_exit") {
+    Type *ArgTypes[] = {Type::getInt8PtrTy(C), Type::getInt8PtrTy(C)};
+
+    Constant *Fn = M.getOrInsertFunction(
+        Func, FunctionType::get(Type::getVoidTy(C), ArgTypes, false));
+
+    Instruction *RetAddr = CallInst::Create(
+        Intrinsic::getDeclaration(&M, Intrinsic::returnaddress),
+        ArrayRef<Value *>(ConstantInt::get(Type::getInt32Ty(C), 0)), "",
+        InsertionPt);
+
+    Value *Args[] = {ConstantExpr::getBitCast(&CurFn, Type::getInt8PtrTy(C)),
+                     RetAddr};
+
+    CallInst::Create(Fn, ArrayRef<Value *>(Args), "", InsertionPt);
+    return;
+  }
+
+  // We only know how to call a fixed set of instrumentation functions, because
+  // they all expect different arguments, etc.
+  report_fatal_error(Twine("Unknown instrumentation function: '") + Func + "'");
+}
+
+static bool runOnFunction(Function &F, bool PostInlining) {
+  StringRef EntryAttr = PostInlining ? "instrument-function-entry-inlined"
+                                     : "instrument-function-entry";
+
+  StringRef ExitAttr = PostInlining ? "instrument-function-exit-inlined"
+                                    : "instrument-function-exit";
+
+  StringRef EntryFunc = F.getFnAttribute(EntryAttr).getValueAsString();
+  StringRef ExitFunc = F.getFnAttribute(ExitAttr).getValueAsString();
+
+  bool Changed = false;
+
+  if (!EntryFunc.empty()) {
+    insertCall(F, EntryFunc, &*F.begin()->getFirstInsertionPt());
+    Changed = true;
+  }
+
+  if (!ExitFunc.empty()) {
+    for (BasicBlock &BB : F) {
+      TerminatorInst *T = BB.getTerminator();
+      if (isa<ReturnInst>(T)) {
+        insertCall(F, ExitFunc, T);
+        Changed = true;
+      }
+    }
+  }
+
+  return Changed;
+}
+
+namespace {
+struct EntryExitInstrumenter : public FunctionPass {
+  static char ID;
+  EntryExitInstrumenter() : FunctionPass(ID) {
+    initializeEntryExitInstrumenterPass(*PassRegistry::getPassRegistry());
+  }
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.addPreserved<GlobalsAAWrapperPass>();
+  }
+  bool runOnFunction(Function &F) override { return ::runOnFunction(F, false); }
+};
+char EntryExitInstrumenter::ID = 0;
+
+struct PostInlineEntryExitInstrumenter : public FunctionPass {
+  static char ID;
+  PostInlineEntryExitInstrumenter() : FunctionPass(ID) {
+    initializePostInlineEntryExitInstrumenterPass(
+        *PassRegistry::getPassRegistry());
+  }
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.addPreserved<GlobalsAAWrapperPass>();
+  }
+  bool runOnFunction(Function &F) override { return ::runOnFunction(F, true); }
+};
+char PostInlineEntryExitInstrumenter::ID = 0;
+}
+
+INITIALIZE_PASS(
+    EntryExitInstrumenter, "ee-instrument",
+    "Instrument function entry/exit with calls to e.g. mcount() (pre inlining)",
+    false, false);
+
+INITIALIZE_PASS(PostInlineEntryExitInstrumenter, "post-inline-ee-instrument",
+                "Instrument function entry/exit with calls to e.g. mcount() "
+                "(post inlining)",
+                false, false);
+
+FunctionPass *llvm::createEntryExitInstrumenterPass() {
+  return new EntryExitInstrumenter();
+}
+
+FunctionPass *llvm::createPostInlineEntryExitInstrumenterPass() {
+  return new PostInlineEntryExitInstrumenter();
+}
+
+PreservedAnalyses
+llvm::EntryExitInstrumenterPass::run(Function &F, FunctionAnalysisManager &AM) {
+  runOnFunction(F, PostInlining);
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
+}

test/CodeGen/PowerPC/mcount-insertion.ll

-; RUN: llc < %s | FileCheck %s
+; RUN: opt -ee-instrument < %s | opt -inline | llc | FileCheck %s
+
+; The run-line mimics how Clang might run the instrumentation passes.
+
 target datalayout = "E-m:e-i64:64-n32:64"
 target triple = "powerpc64-bgq-linux"
 
-define void @test1() #0 {
+
+define void @leaf_function() #0 {
 entry:
   ret void
 
-; CHECK-LABEL: @test1
+; CHECK-LABEL: leaf_function:
 ; CHECK: bl mcount
-; CHECK-NOT: mcount
+; CHECK-NOT: bl
+; CHECK: bl __cyg_profile_func_enter
+; CHECK-NOT: bl
+; CHECK: bl __cyg_profile_func_exit
+; CHECK-NOT: bl
 ; CHECK: blr
 }
 
-attributes #0 = { "counting-function"="mcount" }
 
+define void @root_function() #0 {
+entry:
+  call void @leaf_function()
+  ret void
+
+; CHECK-LABEL: root_function:
+; CHECK: bl mcount
+; CHECK-NOT: bl
+; CHECK: bl __cyg_profile_func_enter
+; CHECK-NOT: bl
+
+; Entry and exit calls, inlined from @leaf_function()
+; CHECK: bl __cyg_profile_func_enter
+; CHECK-NOT: bl
+; CHECK: bl __cyg_profile_func_exit
+; CHECK-NOT: bl
+
+; CHECK: bl __cyg_profile_func_exit
+; CHECK-NOT: bl
+; CHECK: blr
+}
+
+attributes #0 = { "instrument-function-entry-inlined"="mcount" "instrument-function-entry"="__cyg_profile_func_enter" "instrument-function-exit"="__cyg_profile_func_exit" }

test/CodeGen/X86/O0-pipeline.ll

 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Expand Atomic instructions
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       Lower Garbage Collection Instructions
 ; CHECK-NEXT:       Shadow Stack GC Lowering
 ; CHECK-NEXT:       Remove unreachable blocks from the CFG
-; CHECK-NEXT:       Inserts calls to mcount-like functions
+; CHECK-NEXT:       Instrument function entry/exit with calls to e.g. mcount() (post inlining)
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:     Rewrite Symbols
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Exception handling preparation
 ; CHECK-NEXT:       Safe Stack instrumentation pass
 ; CHECK-NEXT:       Insert stack protectors

test/Transforms/CountingFunctionInserter/mcount.ll

-; RUN: opt -S -cfinserter < %s | FileCheck %s
+; RUN: opt -passes="function(ee-instrument),cgscc(inline),function(post-inline-ee-instrument)" -S < %s | FileCheck %s
+
 target datalayout = "E-m:e-i64:64-n32:64"
 target triple = "powerpc64-bgq-linux"
 
-define void @test1() #0 {
+define void @leaf_function() #0 {
 entry:
   ret void
 
-; CHECK-LABEL: define void @test1()
+; CHECK-LABEL: define void @leaf_function()
 ; CHECK: entry:
 ; CHECK-NEXT: call void @mcount()
-; CHECK: ret void
+; CHECK-NEXT: %0 = call i8* @llvm.returnaddress(i32 0)
+; CHECK-NEXT: call void @__cyg_profile_func_enter(i8* bitcast (void ()* @leaf_function to i8*), i8* %0)
+; CHECK-NEXT: %1 = call i8* @llvm.returnaddress(i32 0)
+; CHECK-NEXT: call void @__cyg_profile_func_exit(i8* bitcast (void ()* @leaf_function to i8*), i8* %1)
+; CHECK-NEXT: ret void
 }
 
-define void @test2() #1 {
+
+define void @root_function() #0 {
 entry:
+  call void @leaf_function()
   ret void
 
-; CHECK-LABEL: define void @test2()
+; CHECK-LABEL: define void @root_function()
 ; CHECK: entry:
-; CHECK-NEXT: call void @.mcount()
-; CHECK: ret void
+; CHECK-NEXT: call void @mcount()
+
+; CHECK-NEXT %0 = call i8* @llvm.returnaddress(i32 0)
+; CHECK-NEXT call void @__cyg_profile_func_enter(i8* bitcast (void ()* @root_function to i8*), i8* %0)
+
+; Entry and exit calls, inlined from @leaf_function()
+; CHECK-NEXT %1 = call i8* @llvm.returnaddress(i32 0)
+; CHECK-NEXT call void @__cyg_profile_func_enter(i8* bitcast (void ()* @leaf_function to i8*), i8* %1)
+; CHECK-NEXT %2 = call i8* @llvm.returnaddress(i32 0)
+; CHECK-NEXT call void @__cyg_profile_func_exit(i8* bitcast (void ()* @leaf_function to i8*), i8* %2)
+; CHECK-NEXT %3 = call i8* @llvm.returnaddress(i32 0)
+
+; CHECK-NEXT call void @__cyg_profile_func_exit(i8* bitcast (void ()* @root_function to i8*), i8* %3)
+; CHECK-NEXT ret void
 }
 
-attributes #0 = { "counting-function"="mcount" }
-attributes #1 = { "counting-function"=".mcount" }
+attributes #0 = { "instrument-function-entry-inlined"="mcount" "instrument-function-entry"="__cyg_profile_func_enter" "instrument-function-exit"="__cyg_profile_func_exit" }
+
+
+; The mcount function has many different names.
+
+define void @f1() #1 { entry: ret void }
+; CHECK-LABEL: define void @f1
+; CHECK: call void @.mcount
+
+define void @f2() #2 { entry: ret void }
+; CHECK-LABEL: define void @f2
+; CHECK: call void @"\01__gnu_mcount_nc"
+
+define void @f3() #3 { entry: ret void }
+; CHECK-LABEL: define void @f3
+; CHECK: call void @"\01_mcount"
+
+define void @f4() #4 { entry: ret void }
+; CHECK-LABEL: define void @f4
+; CHECK: call void @"\01mcount"
+
+define void @f5() #5 { entry: ret void }
+; CHECK-LABEL: define void @f5
+; CHECK: call void @__mcount
+
+define void @f6() #6 { entry: ret void }
+; CHECK-LABEL: define void @f6
+; CHECK: call void @_mcount
 
+attributes #1 = { "instrument-function-entry-inlined"=".mcount" }
+attributes #2 = { "instrument-function-entry-inlined"="\01__gnu_mcount_nc" }
+attributes #3 = { "instrument-function-entry-inlined"="\01_mcount" }
+attributes #4 = { "instrument-function-entry-inlined"="\01mcount" }
+attributes #5 = { "instrument-function-entry-inlined"="__mcount" }
+attributes #6 = { "instrument-function-entry-inlined"="_mcount" }

tools/llc/llc.cpp

int main(int argc, char **argv) {
 
   // Initialize codegen and IR passes used by llc so that the -print-after,
   // -print-before, and -stop-after options work.
   PassRegistry *Registry = PassRegistry::getPassRegistry();
   initializeCore(*Registry);
   initializeCodeGen(*Registry);
   initializeLoopStrengthReducePass(*Registry);
   initializeLowerIntrinsicsPass(*Registry);
-  initializeCountingFunctionInserterPass(*Registry);
+  initializeEntryExitInstrumenterPass(*Registry);
+  initializePostInlineEntryExitInstrumenterPass(*Registry);
   initializeUnreachableBlockElimLegacyPassPass(*Registry);
   initializeConstantHoistingLegacyPassPass(*Registry);
   initializeScalarOpts(*Registry);
   initializeVectorization(*Registry);
   initializeScalarizeMaskedMemIntrinPass(*Registry);
   initializeExpandReductionsPass(*Registry);
 
   // Initialize debugging passes.

tools/opt/opt.cpp

int main(int argc, char **argv) {
   initializeRewriteSymbolsLegacyPassPass(Registry);
   initializeWinEHPreparePass(Registry);
   initializeDwarfEHPreparePass(Registry);
   initializeSafeStackLegacyPassPass(Registry);
   initializeSjLjEHPreparePass(Registry);
   initializePreISelIntrinsicLoweringLegacyPassPass(Registry);
   initializeGlobalMergePass(Registry);
   initializeInterleavedAccessPass(Registry);
-  initializeCountingFunctionInserterPass(Registry);
+  initializeEntryExitInstrumenterPass(Registry);
+  initializePostInlineEntryExitInstrumenterPass(Registry);
   initializeUnreachableBlockElimLegacyPassPass(Registry);
   initializeExpandReductionsPass(Registry);
   initializeWriteBitcodePassPass(Registry);
 
 #ifdef LINK_POLLY_INTO_TOOLS
   polly::initializePollyPasses(Registry);
 #endif