[PGO] Make indirect call promotion a utility

[mssimpso]

Summary

This patch factors out the main code transformation utilities in the pgo-driven indirect call promotion pass and places them in Transforms/Utils. The change is intended to be a non-functional change, letting non-pgo-driven passes share a common implementation with the existing pgo-driven pass.

The common utilities are used to conditionally promote indirect call sites to direct call sites. They perform the underlying transformation, and do not consider profile information. The pgo-specific details (e.g., the computation of branch weight metadata) have been left in the indirect call promotion pass.

[davidxl]

Can you make this patch a pure NFC one and split out the enhancement part into another patch?

[mssimpso]

In D40658#940668, @davidxl wrote:

Can you make this patch a pure NFC one and split out the enhancement part into another patch?

Will do. Thanks, David.

[mssimpso]

Made this patch NFC and split out the functional changes.

[davidxl]

lgtm

[davide]

I'm fine with this but David Li (or other PGO people) should take a look as well. Thanks.

include/llvm/Transforms/Instrumentation.h

 // PGO Instrumention
 ModulePass *createPGOInstrumentationGenLegacyPass();
 ModulePass *
 createPGOInstrumentationUseLegacyPass(StringRef Filename = StringRef(""));
 ModulePass *createPGOIndirectCallPromotionLegacyPass(bool InLTO = false,
                                                      bool SamplePGO = false);
 FunctionPass *createPGOMemOPSizeOptLegacyPass();
 
-// Helper function to check if it is legal to promote indirect call \p Inst
-// to a direct call of function \p F. Stores the reason in \p Reason.
-bool isLegalToPromote(Instruction *Inst, Function *F, const char **Reason);
+// The pgo-specific indirect call promotion function declared below is used by
+// the pgo-driven indirect call promotion and sample profile passes. It's a
+// wrapper around llvm::promoteCall, et al. that additionally computes !prof
+// metadata. We place it in a pgo namespace so it's not confused with the
+// generic utilities.
+namespace pgo {
 
 // Helper function that transforms Inst (either an indirect-call instruction, or
 // an invoke instruction , to a conditional call to F. This is like:
 //     if (Inst.CalledValue == F)
 //        F(...);
 //     else
 //        Inst(...);
 //     end
 // TotalCount is the profile count value that the instruction executes.
 // Count is the profile count value that F is the target function.
 // These two values are used to update the branch weight.
 // If \p AttachProfToDirectCall is true, a prof metadata is attached to the
 // new direct call to contain \p Count.
 // Returns the promoted direct call instruction.
 Instruction *promoteIndirectCall(Instruction *Inst, Function *F, uint64_t Count,
                                  uint64_t TotalCount,
                                  bool AttachProfToDirectCall,
                                  OptimizationRemarkEmitter *ORE);
+} // namespace pgo
 
 /// Options for the frontend instrumentation based profiling pass.
 struct InstrProfOptions {
   // Add the 'noredzone' attribute to added runtime library calls.
   bool NoRedZone = false;
 
   // Do counter register promotion
   bool DoCounterPromotion = false;

include/llvm/Transforms/Utils/CallPromotionUtils.h

+//===- CallPromotionUtils.h - Utilities for call promotion ------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares utilities useful for promoting indirect call sites to
+// direct call sites.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TRANSFORMS_UTILS_CALLPROMOTIONUTILS_H
+#define LLVM_TRANSFORMS_UTILS_CALLPROMOTIONUTILS_H
+
+#include "llvm/IR/InstrTypes.h"
+
+namespace llvm {
+
+class CallSite;
+
+/// Return true if the given indirect call site can be made to call \p Callee.
+///
+/// This function ensures that the number and type of the call site's arguments
+/// and return value match those of the given function. If the types do not
+/// match exactly, they must at least be bitcast compatible. If \p FailureReason
+/// is non-null and the indirect call cannot be promoted, the failure reason
+/// will be stored in it.
+bool isLegalToPromote(CallSite CS, Function *Callee,
+                      const char **FailureReason = nullptr);
+
+/// Promote the given indirect call site to unconditionally call \p Callee.
+///
+/// If the function type of the call site doesn't match that of the callee,
+/// bitcast instructions are inserted where appropriate. If \p Casts is
+/// non-null, these bitcasts are collected in the provided container.
+void promoteCall(CallSite CS, Function *Callee,
+                 SmallVectorImpl<CastInst *> *Casts = nullptr);
+
+/// Demote the given direct call site to call \p CalledValue.
+///
+/// This function is meant to be used in conjunction with \p promoteCall. As
+/// such, any bitcast instructions contained in \p Casts, which were created by
+/// \p promoteCall, will be erased. Together, these functions can be used to
+/// temporarily promote a call to facilitate analyzing the program as if the
+/// call were direct.
+void demoteCall(CallSite CS, Value *CalledValue, ArrayRef<CastInst *> Casts);
+
+/// Promote the given indirect call site to conditionally call \p Callee.
+///
+/// This function creates an if-then-else structure at the location of the call
+/// site. The original call site is promoted and moved into the "then" block. A
+/// clone of the indirect call site is placed in the "else" block and returned.
+/// If \p BranchWeights is non-null, it will be used to set !prof metadata on
+/// the new conditional branch.
+Instruction *promoteCallWithIfThenElse(CallSite CS, Function *Callee,
+                                       MDNode *BranchWeights = nullptr);
+
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_UTILS_CALLPROMOTIONUTILS_H

lib/Transforms/IPO/SampleProfile.cpp

 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ErrorOr.h"
 #include "llvm/Support/GenericDomTree.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/IPO.h"
 #include "llvm/Transforms/Instrumentation.h"
+#include "llvm/Transforms/Utils/CallPromotionUtils.h"
 #include "llvm/Transforms/Utils/Cloning.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <functional>
 #include <limits>
 #include <map>
 #include <memory>
for (auto I : CIS) {
           if (CalleeFunctionName == F.getName())
             continue;
 
           const char *Reason = "Callee function not available";
           auto R = SymbolMap.find(CalleeFunctionName);
           if (R != SymbolMap.end() && R->getValue() &&
               !R->getValue()->isDeclaration() &&
               R->getValue()->getSubprogram() &&
-              isLegalToPromote(I, R->getValue(), &Reason)) {
+              isLegalToPromote(CallSite(I), R->getValue(), &Reason)) {
             uint64_t C = FS->getEntrySamples();
-            Instruction *DI = promoteIndirectCall(
-                I, R->getValue(), C, Sum, false, ORE);
+            Instruction *DI = I;
+            I = pgo::promoteIndirectCall(I, R->getValue(), C, Sum, false, ORE);
             Sum -= C;
             PromotedInsns.insert(I);
             // If profile mismatches, we should not attempt to inline DI.
             if ((isa<CallInst>(DI) || isa<InvokeInst>(DI)) &&
                 inlineCallInstruction(DI))
               LocalChanged = true;
           } else {
             DEBUG(dbgs()

lib/Transforms/Instrumentation/IndirectCallPromotion.cpp

 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/Instrumentation.h"
 #include "llvm/Transforms/PGOInstrumentation.h"
 #include "llvm/Transforms/Utils/BasicBlockUtils.h"
+#include "llvm/Transforms/Utils/CallPromotionUtils.h"
 #include <cassert>
 #include <cstdint>
 #include <memory>
 #include <string>
 #include <utility>
 #include <vector>
 
 using namespace llvm;
private:
   // TotalCount is the total profiled count of call executions, and
   // NumCandidates is the number of candidate entries in ValueDataRef.
   std::vector<PromotionCandidate> getPromotionCandidatesForCallSite(
       Instruction *Inst, const ArrayRef<InstrProfValueData> &ValueDataRef,
       uint64_t TotalCount, uint32_t NumCandidates);
 
   // Promote a list of targets for one indirect-call callsite. Return
   // the number of promotions.
-  uint32_t tryToPromote(Instruction *Inst,
+  uint32_t tryToPromote(Instruction *&Inst,
                         const std::vector<PromotionCandidate> &Candidates,
                         uint64_t &TotalCount);
 
 public:
   ICallPromotionFunc(Function &Func, Module *Modu, InstrProfSymtab *Symtab,
                      bool SamplePGO, OptimizationRemarkEmitter &ORE)
       : F(Func), M(Modu), Symtab(Symtab), SamplePGO(SamplePGO), ORE(ORE) {}
   ICallPromotionFunc(const ICallPromotionFunc &) = delete;
   ICallPromotionFunc &operator=(const ICallPromotionFunc &) = delete;
 
   bool processFunction(ProfileSummaryInfo *PSI);
 };
 
 } // end anonymous namespace
 
-bool llvm::isLegalToPromote(Instruction *Inst, Function *F,
-                            const char **Reason) {
-  // Check the return type.
-  Type *CallRetType = Inst->getType();
-  if (!CallRetType->isVoidTy()) {
-    Type *FuncRetType = F->getReturnType();
-    if (FuncRetType != CallRetType &&
-        !CastInst::isBitCastable(FuncRetType, CallRetType)) {
-      if (Reason)
-        *Reason = "Return type mismatch";
-      return false;
-    }
-  }
-
-  // Check if the arguments are compatible with the parameters
-  FunctionType *DirectCalleeType = F->getFunctionType();
-  unsigned ParamNum = DirectCalleeType->getFunctionNumParams();
-  CallSite CS(Inst);
-  unsigned ArgNum = CS.arg_size();
-
-  if (ParamNum != ArgNum && !DirectCalleeType->isVarArg()) {
-    if (Reason)
-      *Reason = "The number of arguments mismatch";
-    return false;
-  }
-
-  for (unsigned I = 0; I < ParamNum; ++I) {
-    Type *PTy = DirectCalleeType->getFunctionParamType(I);
-    Type *ATy = CS.getArgument(I)->getType();
-    if (PTy == ATy)
-      continue;
-    if (!CastInst::castIsValid(Instruction::BitCast, CS.getArgument(I), PTy)) {
-      if (Reason)
-        *Reason = "Argument type mismatch";
-      return false;
-    }
-  }
-
-  DEBUG(dbgs() << " #" << NumOfPGOICallPromotion << " Promote the icall to "
-               << F->getName() << "\n");
-  return true;
-}
-
 // Indirect-call promotion heuristic. The direct targets are sorted based on
 // the count. Stop at the first target that is not promoted.
 std::vector<ICallPromotionFunc::PromotionCandidate>
 ICallPromotionFunc::getPromotionCandidatesForCallSite(
     Instruction *Inst, const ArrayRef<InstrProfValueData> &ValueDataRef,
     uint64_t TotalCount, uint32_t NumCandidates) {
   std::vector<PromotionCandidate> Ret;
 
if (TargetFunction == nullptr) {
       ORE.emit([&]() {
         return OptimizationRemarkMissed(DEBUG_TYPE, "UnableToFindTarget", Inst)
                << "Cannot promote indirect call: target not found";
       });
       break;
     }
 
     const char *Reason = nullptr;
-    if (!isLegalToPromote(Inst, TargetFunction, &Reason)) {
+    if (!isLegalToPromote(CallSite(Inst), TargetFunction, &Reason)) {
       using namespace ore;
 
       ORE.emit([&]() {
         return OptimizationRemarkMissed(DEBUG_TYPE, "UnableToPromote", Inst)
                << "Cannot promote indirect call to "
                << NV("TargetFunction", TargetFunction) << " with count of "
                << NV("Count", Count) << ": " << Reason;
       });
       break;
     }
 
     Ret.push_back(PromotionCandidate(TargetFunction, Count));
     TotalCount -= Count;
   }
   return Ret;
 }
 
-// Create a diamond structure for If_Then_Else. Also update the profile
-// count. Do the fix-up for the invoke instruction.
-static void createIfThenElse(Instruction *Inst, Function *DirectCallee,
+Instruction *llvm::pgo::promoteIndirectCall(Instruction *Inst,
+                                            Function *DirectCallee,
                                             uint64_t Count, uint64_t TotalCount,
-                             BasicBlock **DirectCallBB,
-                             BasicBlock **IndirectCallBB,
-                             BasicBlock **MergeBB) {
-  CallSite CS(Inst);
-  Value *OrigCallee = CS.getCalledValue();
-
-  IRBuilder<> BBBuilder(Inst);
-  LLVMContext &Ctx = Inst->getContext();
-  Value *BCI1 =
-      BBBuilder.CreateBitCast(OrigCallee, Type::getInt8PtrTy(Ctx), "");
-  Value *BCI2 =
-      BBBuilder.CreateBitCast(DirectCallee, Type::getInt8PtrTy(Ctx), "");
-  Value *PtrCmp = BBBuilder.CreateICmpEQ(BCI1, BCI2, "");
+                                            bool AttachProfToDirectCall,
+                                            OptimizationRemarkEmitter *ORE) {
 
   uint64_t ElseCount = TotalCount - Count;
   uint64_t MaxCount = (Count >= ElseCount ? Count : ElseCount);
   uint64_t Scale = calculateCountScale(MaxCount);
   MDBuilder MDB(Inst->getContext());
   MDNode *BranchWeights = MDB.createBranchWeights(
       scaleBranchCount(Count, Scale), scaleBranchCount(ElseCount, Scale));
-  TerminatorInst *ThenTerm, *ElseTerm;
-  SplitBlockAndInsertIfThenElse(PtrCmp, Inst, &ThenTerm, &ElseTerm,
-                                BranchWeights);
-  *DirectCallBB = ThenTerm->getParent();
-  (*DirectCallBB)->setName("if.true.direct_targ");
-  *IndirectCallBB = ElseTerm->getParent();
-  (*IndirectCallBB)->setName("if.false.orig_indirect");
-  *MergeBB = Inst->getParent();
-  (*MergeBB)->setName("if.end.icp");
-
-  // Special handing of Invoke instructions.
-  InvokeInst *II = dyn_cast<InvokeInst>(Inst);
-  if (!II)
-    return;
-
-  // We don't need branch instructions for invoke.
-  ThenTerm->eraseFromParent();
-  ElseTerm->eraseFromParent();
-
-  // Add jump from Merge BB to the NormalDest. This is needed for the newly
-  // created direct invoke stmt -- as its NormalDst will be fixed up to MergeBB.
-  BranchInst::Create(II->getNormalDest(), *MergeBB);
-}
-
-// Find the PHI in BB that have the CallResult as the operand.
-static bool getCallRetPHINode(BasicBlock *BB, Instruction *Inst) {
-  BasicBlock *From = Inst->getParent();
-  for (auto &I : *BB) {
-    PHINode *PHI = dyn_cast<PHINode>(&I);
-    if (!PHI)
-      continue;
-    int IX = PHI->getBasicBlockIndex(From);
-    if (IX == -1)
-      continue;
-    Value *V = PHI->getIncomingValue(IX);
-    if (dyn_cast<Instruction>(V) == Inst)
-      return true;
-  }
-  return false;
-}
-
-// This method fixes up PHI nodes in BB where BB is the UnwindDest of an
-// invoke instruction. In BB, there may be PHIs with incoming block being
-// OrigBB (the MergeBB after if-then-else splitting). After moving the invoke
-// instructions to its own BB, OrigBB is no longer the predecessor block of BB.
-// Instead two new predecessors are added: IndirectCallBB and DirectCallBB,
-// so the PHI node's incoming BBs need to be fixed up accordingly.
-static void fixupPHINodeForUnwind(Instruction *Inst, BasicBlock *BB,
-                                  BasicBlock *OrigBB,
-                                  BasicBlock *IndirectCallBB,
-                                  BasicBlock *DirectCallBB) {
-  for (auto &I : *BB) {
-    PHINode *PHI = dyn_cast<PHINode>(&I);
-    if (!PHI)
-      continue;
-    int IX = PHI->getBasicBlockIndex(OrigBB);
-    if (IX == -1)
-      continue;
-    Value *V = PHI->getIncomingValue(IX);
-    PHI->addIncoming(V, IndirectCallBB);
-    PHI->setIncomingBlock(IX, DirectCallBB);
-  }
-}
 
-// This method fixes up PHI nodes in BB where BB is the NormalDest of an
-// invoke instruction. In BB, there may be PHIs with incoming block being
-// OrigBB (the MergeBB after if-then-else splitting). After moving the invoke
-// instructions to its own BB, a new incoming edge will be added to the original
-// NormalDstBB from the IndirectCallBB.
-static void fixupPHINodeForNormalDest(Instruction *Inst, BasicBlock *BB,
-                                      BasicBlock *OrigBB,
-                                      BasicBlock *IndirectCallBB,
-                                      Instruction *NewInst) {
-  for (auto &I : *BB) {
-    PHINode *PHI = dyn_cast<PHINode>(&I);
-    if (!PHI)
-      continue;
-    int IX = PHI->getBasicBlockIndex(OrigBB);
-    if (IX == -1)
-      continue;
-    Value *V = PHI->getIncomingValue(IX);
-    if (dyn_cast<Instruction>(V) == Inst) {
-      PHI->setIncomingBlock(IX, IndirectCallBB);
-      PHI->addIncoming(NewInst, OrigBB);
-      continue;
-    }
-    PHI->addIncoming(V, IndirectCallBB);
-  }
-}
-
-// Add a bitcast instruction to the direct-call return value if needed.
-static Instruction *insertCallRetCast(const Instruction *Inst,
-                                      Instruction *DirectCallInst,
-                                      Function *DirectCallee) {
-  if (Inst->getType()->isVoidTy())
-    return DirectCallInst;
-
-  Type *CallRetType = Inst->getType();
-  Type *FuncRetType = DirectCallee->getReturnType();
-  if (FuncRetType == CallRetType)
-    return DirectCallInst;
-
-  BasicBlock *InsertionBB;
-  if (CallInst *CI = dyn_cast<CallInst>(DirectCallInst))
-    InsertionBB = CI->getParent();
-  else
-    InsertionBB = (dyn_cast<InvokeInst>(DirectCallInst))->getNormalDest();
-
-  return (new BitCastInst(DirectCallInst, CallRetType, "",
-                          InsertionBB->getTerminator()));
-}
-
-// Create a DirectCall instruction in the DirectCallBB.
-// Parameter Inst is the indirect-call (invoke) instruction.
-// DirectCallee is the decl of the direct-call (invoke) target.
-// DirecallBB is the BB that the direct-call (invoke) instruction is inserted.
-// MergeBB is the bottom BB of the if-then-else-diamond after the
-// transformation. For invoke instruction, the edges from DirectCallBB and
-// IndirectCallBB to MergeBB are removed before this call (during
-// createIfThenElse). Stores the pointer to the Instruction that cast
-// the direct call in \p CastInst.
-static Instruction *createDirectCallInst(const Instruction *Inst,
-                                         Function *DirectCallee,
-                                         BasicBlock *DirectCallBB,
-                                         BasicBlock *MergeBB,
-                                         Instruction *&CastInst) {
-  Instruction *NewInst = Inst->clone();
-  if (CallInst *CI = dyn_cast<CallInst>(NewInst)) {
-    CI->setCalledFunction(DirectCallee);
-    CI->mutateFunctionType(DirectCallee->getFunctionType());
-  } else {
-    // Must be an invoke instruction. Direct invoke's normal destination is
-    // fixed up to MergeBB. MergeBB is the place where return cast is inserted.
-    // Also since IndirectCallBB does not have an edge to MergeBB, there is no
-    // need to insert new PHIs into MergeBB.
-    InvokeInst *II = dyn_cast<InvokeInst>(NewInst);
-    assert(II);
-    II->setCalledFunction(DirectCallee);
-    II->mutateFunctionType(DirectCallee->getFunctionType());
-    II->setNormalDest(MergeBB);
-  }
-
-  DirectCallBB->getInstList().insert(DirectCallBB->getFirstInsertionPt(),
-                                     NewInst);
-
-  // Clear the value profile data.
-  NewInst->setMetadata(LLVMContext::MD_prof, nullptr);
-  CallSite NewCS(NewInst);
-  FunctionType *DirectCalleeType = DirectCallee->getFunctionType();
-  unsigned ParamNum = DirectCalleeType->getFunctionNumParams();
-  for (unsigned I = 0; I < ParamNum; ++I) {
-    Type *ATy = NewCS.getArgument(I)->getType();
-    Type *PTy = DirectCalleeType->getParamType(I);
-    if (ATy != PTy) {
-      BitCastInst *BI = new BitCastInst(NewCS.getArgument(I), PTy, "", NewInst);
-      NewCS.setArgument(I, BI);
-    }
-  }
-
-  CastInst = insertCallRetCast(Inst, NewInst, DirectCallee);
-  return NewInst;
-}
-
-// Create a PHI to unify the return values of calls.
-static void insertCallRetPHI(Instruction *Inst, Instruction *CallResult,
-                             Function *DirectCallee) {
-  if (Inst->getType()->isVoidTy())
-    return;
-
-  if (Inst->use_empty())
-    return;
-
-  BasicBlock *RetValBB = CallResult->getParent();
-
-  BasicBlock *PHIBB;
-  if (InvokeInst *II = dyn_cast<InvokeInst>(CallResult))
-    RetValBB = II->getNormalDest();
-
-  PHIBB = RetValBB->getSingleSuccessor();
-  if (getCallRetPHINode(PHIBB, Inst))
-    return;
-
-  PHINode *CallRetPHI = PHINode::Create(Inst->getType(), 0);
-  PHIBB->getInstList().push_front(CallRetPHI);
-  Inst->replaceAllUsesWith(CallRetPHI);
-  CallRetPHI->addIncoming(Inst, Inst->getParent());
-  CallRetPHI->addIncoming(CallResult, RetValBB);
-}
-
-// This function does the actual indirect-call promotion transformation:
-// For an indirect-call like:
-//     Ret = (*Foo)(Args);
-// It transforms to:
-//     if (Foo == DirectCallee)
-//        Ret1 = DirectCallee(Args);
-//     else
-//        Ret2 = (*Foo)(Args);
-//     Ret = phi(Ret1, Ret2);
-// It adds type casts for the args do not match the parameters and the return
-// value. Branch weights metadata also updated.
-// If \p AttachProfToDirectCall is true, a prof metadata is attached to the
-// new direct call to contain \p Count. This is used by SamplePGO inliner to
-// check callsite hotness.
-// Returns the promoted direct call instruction.
-Instruction *llvm::promoteIndirectCall(Instruction *Inst,
-                                       Function *DirectCallee, uint64_t Count,
-                                       uint64_t TotalCount,
-                                       bool AttachProfToDirectCall,
-                                       OptimizationRemarkEmitter *ORE) {
-  assert(DirectCallee != nullptr);
-  BasicBlock *BB = Inst->getParent();
-  // Just to suppress the non-debug build warning.
-  (void)BB;
-  DEBUG(dbgs() << "\n\n== Basic Block Before ==\n");
-  DEBUG(dbgs() << *BB << "\n");
-
-  BasicBlock *DirectCallBB, *IndirectCallBB, *MergeBB;
-  createIfThenElse(Inst, DirectCallee, Count, TotalCount, &DirectCallBB,
-                   &IndirectCallBB, &MergeBB);
-
-  // If the return type of the NewInst is not the same as the Inst, a CastInst
-  // is needed for type casting. Otherwise CastInst is the same as NewInst.
-  Instruction *CastInst = nullptr;
   Instruction *NewInst =
-      createDirectCallInst(Inst, DirectCallee, DirectCallBB, MergeBB, CastInst);
+      promoteCallWithIfThenElse(CallSite(Inst), DirectCallee, BranchWeights);
 
   if (AttachProfToDirectCall) {
     SmallVector<uint32_t, 1> Weights;
     Weights.push_back(Count);
-    MDBuilder MDB(NewInst->getContext());
-    NewInst->setMetadata(LLVMContext::MD_prof, MDB.createBranchWeights(Weights));
-  }
-
-  // Move Inst from MergeBB to IndirectCallBB.
-  Inst->removeFromParent();
-  IndirectCallBB->getInstList().insert(IndirectCallBB->getFirstInsertionPt(),
-                                       Inst);
-
-  if (InvokeInst *II = dyn_cast<InvokeInst>(Inst)) {
-    // At this point, the original indirect invoke instruction has the original
-    // UnwindDest and NormalDest. For the direct invoke instruction, the
-    // NormalDest points to MergeBB, and MergeBB jumps to the original
-    // NormalDest. MergeBB might have a new bitcast instruction for the return
-    // value. The PHIs are with the original NormalDest. Since we now have two
-    // incoming edges to NormalDest and UnwindDest, we have to do some fixups.
-    //
-    // UnwindDest will not use the return value. So pass nullptr here.
-    fixupPHINodeForUnwind(Inst, II->getUnwindDest(), MergeBB, IndirectCallBB,
-                          DirectCallBB);
-    // We don't need to update the operand from NormalDest for DirectCallBB.
-    // Pass nullptr here.
-    fixupPHINodeForNormalDest(Inst, II->getNormalDest(), MergeBB,
-                              IndirectCallBB, CastInst);
+    MDBuilder MDB(Inst->getContext());
+    Inst->setMetadata(LLVMContext::MD_prof, MDB.createBranchWeights(Weights));
   }
 
-  insertCallRetPHI(Inst, CastInst, DirectCallee);
-
-  DEBUG(dbgs() << "\n== Basic Blocks After ==\n");
-  DEBUG(dbgs() << *BB << *DirectCallBB << *IndirectCallBB << *MergeBB << "\n");
-
   using namespace ore;
 
   if (ORE)
     ORE->emit([&]() {
       return OptimizationRemark(DEBUG_TYPE, "Promoted", Inst)
              << "Promote indirect call to " << NV("DirectCallee", DirectCallee)
              << " with count " << NV("Count", Count) << " out of "
              << NV("TotalCount", TotalCount);
     });
   return NewInst;
 }
 
 // Promote indirect-call to conditional direct-call for one callsite.
 uint32_t ICallPromotionFunc::tryToPromote(
-    Instruction *Inst, const std::vector<PromotionCandidate> &Candidates,
+    Instruction *&Inst, const std::vector<PromotionCandidate> &Candidates,
     uint64_t &TotalCount) {
   uint32_t NumPromoted = 0;
 
   for (auto &C : Candidates) {
     uint64_t Count = C.Count;
-    promoteIndirectCall(Inst, C.TargetFunction, Count, TotalCount, SamplePGO,
-                        &ORE);
+    Inst = pgo::promoteIndirectCall(Inst, C.TargetFunction, Count, TotalCount,
+                                    SamplePGO, &ORE);
     assert(TotalCount >= Count);
     TotalCount -= Count;
     NumOfPGOICallPromotion++;
     NumPromoted++;
   }
   return NumPromoted;
 }
 

lib/Transforms/Utils/CMakeLists.txt

 add_llvm_library(LLVMTransformUtils
   ASanStackFrameLayout.cpp
   AddDiscriminators.cpp
   BasicBlockUtils.cpp
   BreakCriticalEdges.cpp
   BuildLibCalls.cpp
   BypassSlowDivision.cpp
+  CallPromotionUtils.cpp
   CloneFunction.cpp
   CloneModule.cpp
   CodeExtractor.cpp
   CtorUtils.cpp
   DemoteRegToStack.cpp
   EntryExitInstrumenter.cpp
   EscapeEnumerator.cpp
   Evaluator.cpp

lib/Transforms/Utils/CallPromotionUtils.cpp

+//===- CallPromotionUtils.cpp - Utilities for call promotion ----*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements utilities useful for promoting indirect call sites to
+// direct call sites.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Transforms/Utils/CallPromotionUtils.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/IR/CallSite.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "call-promotion-utils"
+
+/// Fix-up phi nodes in an invoke instruction's normal destination.
+///
+/// After versioning an invoke instruction, values coming from the original
+/// block will now be coming from the "merge" block.
+static void fixupPHINodeForNormalDest(InvokeInst *Invoke, BasicBlock *OrigBlock,
+                                      BasicBlock *MergeBlock) {
+  for (auto &I : *Invoke->getNormalDest()) {
+    auto *Phi = dyn_cast<PHINode>(&I);
+    if (!Phi)
+      break;
+    int Idx = Phi->getBasicBlockIndex(OrigBlock);
+    if (Idx == -1)
+      continue;
+    Phi->setIncomingBlock(Idx, MergeBlock);
+  }
+}
+
+/// Fix-up phi nodes in an invoke instruction's unwind destination.
+///
+/// After versioning an invoke instruction, values coming from the original
+/// block will now be coming from either the "then" block or the "else" block.
+static void fixupPHINodeForUnwindDest(InvokeInst *Invoke, BasicBlock *OrigBlock,
+                                      BasicBlock *ThenBlock,
+                                      BasicBlock *ElseBlock) {
+  for (auto &I : *Invoke->getUnwindDest()) {
+    auto *Phi = dyn_cast<PHINode>(&I);
+    if (!Phi)
+      break;
+    int Idx = Phi->getBasicBlockIndex(OrigBlock);
+    if (Idx == -1)
+      continue;
+    auto *V = Phi->getIncomingValue(Idx);
+    Phi->setIncomingBlock(Idx, ThenBlock);
+    Phi->addIncoming(V, ElseBlock);
+  }
+}
+
+/// Create a phi node for the returned value of a call or invoke instruction.
+///
+/// After versioning a call or invoke instruction that returns a value, we have
+/// to merge the value of the original and new instructions. We do this by
+/// creating a phi node at the front of the "merge" block, and replacing uses of
+/// the original instruction with this phi node.
+static void createRetPHINode(Instruction *OrigInst, Instruction *NewInst,
+                             BasicBlock *ThenBlock, BasicBlock *ElseBlock,
+                             BasicBlock *MergeBlock, IRBuilder<> &Builder) {
+  if (!OrigInst->getType()->isVoidTy() && !OrigInst->use_empty()) {
+    Builder.SetInsertPoint(&MergeBlock->front());
+    PHINode *Phi = Builder.CreatePHI(NewInst->getType(), 0);
+    SmallVector<User *, 16> UsersToUpdate;
+    for (User *U : OrigInst->users())
+      UsersToUpdate.push_back(U);
+    for (User *U : UsersToUpdate)
+      U->replaceUsesOfWith(OrigInst, Phi);
+    Phi->addIncoming(NewInst, ElseBlock);
+    Phi->addIncoming(OrigInst, ThenBlock);
+  }
+}
+
+/// Cast a call or invoke instruction to the given type.
+///
+/// When promoting a call site, the return type of the call site might not match
+/// that of the callee. If this is the case, we have to cast the returned value
+/// to the correct type. The location of the cast depends of if we have a call
+/// or invoke instruction.
+static void createRetBitCast(CallSite CS, Type *RetTy,
+                             SmallVectorImpl<CastInst *> *Casts) {
+
+  // Save the users of the calling instruction. These uses will be changed to
+  // use the bitcast after we create it.
+  SmallVector<User *, 16> UsersToUpdate;
+  for (User *U : CS.getInstruction()->users())
+    UsersToUpdate.push_back(U);
+
+  // Determine an appropriate location to create the bitcast for the return
+  // value. The location depends on if we have a call or invoke instruction.
+  Instruction *InsertBefore = nullptr;
+  if (auto *Invoke = dyn_cast<InvokeInst>(CS.getInstruction()))
+    InsertBefore =
+        &SplitEdge(Invoke->getParent(), Invoke->getNormalDest())->front();
+  else
+    InsertBefore = &*std::next(CS.getInstruction()->getIterator());
+
+  // Bitcast the return value to the correct type.
+  auto *Cast = CastInst::Create(Instruction::BitCast, CS.getInstruction(),
+                                RetTy, "", InsertBefore);
+  if (Casts)
+    Casts->push_back(Cast);
+
+  // Replace all the original uses of the calling instruction with the bitcast.
+  for (User *U : UsersToUpdate)
+    U->replaceUsesOfWith(CS.getInstruction(), Cast);
+}
+
+/// Predicate and clone the given call site.
+///
+/// This function creates an if-then-else structure at the location of the call
+/// site. The "if" condition compares the call site's called value to the given
+/// callee. The original call site is moved into the "then" block, and a clone
+/// of the call site is placed in the "else" block. The cloned instruction is
+/// returned. If the call site happens to be an invoke instruction, further work
+/// is done to fix phi nodes in the invoke's normal and unwind destinations.
+static Instruction *versionCallSite(CallSite CS, Value *Callee,
+                                    MDNode *BranchWeights) {
+
+  IRBuilder<> Builder(CS.getInstruction());
+  Instruction *OrigInst = CS.getInstruction();
+  BasicBlock *OrigBlock = OrigInst->getParent();
+
+  // Create the compare. The called value and callee must have the same type to
+  // be compared. If the don't, cast the callee to have the called value's type.
+  if (CS.getCalledValue()->getType() != Callee->getType())
+    Callee = Builder.CreateBitCast(Callee, CS.getCalledValue()->getType());
+  auto *Cond = Builder.CreateICmpEQ(CS.getCalledValue(), Callee);
+
+  // Create an if-then-else structure. The original instruction is moved into
+  // the "then" block, and a clone of the original instruction is placed in the
+  // "else" block.
+  TerminatorInst *ThenTerm = nullptr;
+  TerminatorInst *ElseTerm = nullptr;
+  SplitBlockAndInsertIfThenElse(Cond, CS.getInstruction(), &ThenTerm, &ElseTerm,
+                                BranchWeights);
+  BasicBlock *ThenBlock = ThenTerm->getParent();
+  BasicBlock *ElseBlock = ElseTerm->getParent();
+  BasicBlock *MergeBlock = OrigInst->getParent();
+
+  ThenBlock->setName("if.true.direct_targ");
+  ElseBlock->setName("if.false.orig_indirect");
+  MergeBlock->setName("if.end.icp");
+
+  Instruction *NewInst = OrigInst->clone();
+  OrigInst->moveBefore(ThenTerm);
+  NewInst->insertBefore(ElseTerm);
+
+  // If the original call site is an invoke instruction, we have extra work to
+  // do since invoke instructions are terminating. We have to fix-up phi nodes
+  // in the invoke's normal and unwind destinations.
+  if (auto *OrigInvoke = dyn_cast<InvokeInst>(OrigInst)) {
+    auto *NewInvoke = cast<InvokeInst>(NewInst);
+
+    // Invoke instructions are terminating, so we don't need the terminator
+    // instructions that were just created.
+    ThenTerm->eraseFromParent();
+    ElseTerm->eraseFromParent();
+
+    // Branch from the "merge" block to the original normal destination.
+    Builder.SetInsertPoint(MergeBlock);
+    Builder.CreateBr(OrigInvoke->getNormalDest());
+
+    // Fix-up phi nodes in the original invoke's normal and unwind destinations.
+    fixupPHINodeForNormalDest(OrigInvoke, OrigBlock, MergeBlock);
+    fixupPHINodeForUnwindDest(OrigInvoke, OrigBlock, ThenBlock, ElseBlock);
+
+    // Now set the normal destinations of the invoke instructions to be the
+    // "merge" block.
+    OrigInvoke->setNormalDest(MergeBlock);
+    NewInvoke->setNormalDest(MergeBlock);
+  }
+
+  // Create a phi node for the returned value of the call site.
+  createRetPHINode(OrigInst, NewInst, ThenBlock, ElseBlock, MergeBlock,
+                   Builder);
+
+  return NewInst;
+}
+
+bool llvm::isLegalToPromote(CallSite CS, Function *Callee,
+                            const char **FailureReason) {
+  assert(!CS.getCalledFunction() && "Only indirect call sites can be promoted");
+
+  // Check the return type. The callee's return value type must be bitcast
+  // compatible with the call site's type.
+  Type *CallRetTy = CS.getInstruction()->getType();
+  Type *FuncRetTy = Callee->getReturnType();
+  if (CallRetTy != FuncRetTy)
+    if (!CastInst::isBitCastable(FuncRetTy, CallRetTy)) {
+      if (FailureReason)
+        *FailureReason = "Return type mismatch";
+      return false;
+    }
+
+  // The number of formal arguments of the callee.
+  unsigned NumParams = Callee->getFunctionType()->getNumParams();
+
+  // Check the number of arguments. The callee and call site must agree on the
+  // number of arguments.
+  if (CS.arg_size() != NumParams && !Callee->isVarArg()) {
+    if (FailureReason)
+      *FailureReason = "Argument number mismatch";
+    return false;
+  }
+
+  // Check the argument types. The callee's formal argument types must be
+  // bitcast compatible with the corresponding actual argument types of the call
+  // site.
+  for (unsigned I = 0; I < NumParams; ++I) {
+    Type *FormalTy = Callee->getFunctionType()->getFunctionParamType(I);
+    Type *ActualTy = CS.getArgument(I)->getType();
+    if (FormalTy == ActualTy)
+      continue;
+    if (!CastInst::isBitCastable(ActualTy, FormalTy)) {
+      if (FailureReason)
+        *FailureReason = "Argument type mismatch";
+      return false;
+    }
+  }
+
+  return true;
+}
+
+void llvm::promoteCall(CallSite CS, Function *Callee,
+                       SmallVectorImpl<CastInst *> *Casts) {
+  assert(!CS.getCalledFunction() && "Only indirect call sites can be promoted");
+
+  // Set the called function of the call site to be the given callee.
+  CS.setCalledFunction(Callee);
+
+  // Since the call site will no longer be direct, we must clear metadata that
+  // is only appropriate for indirect calls. This includes !prof and !callees
+  // metadata.
+  CS.getInstruction()->setMetadata(LLVMContext::MD_prof, nullptr);
+  CS.getInstruction()->setMetadata(LLVMContext::MD_callees, nullptr);
+
+  // If the function type of the call site matches that of the callee, no
+  // additional work is required.
+  if (CS.getFunctionType() == Callee->getFunctionType())
+    return;
+
+  // Save the return types of the call site and callee.
+  Type *CallSiteRetTy = CS.getInstruction()->getType();
+  Type *CalleeRetTy = Callee->getReturnType();
+
+  // Change the function type of the call site the match that of the callee.
+  CS.mutateFunctionType(Callee->getFunctionType());
+
+  // Inspect the arguments of the call site. If an argument's type doesn't
+  // match the corresponding formal argument's type in the callee, bitcast it
+  // to the correct type.
+  for (Use &U : CS.args()) {
+    unsigned ArgNo = CS.getArgumentNo(&U);
+    Type *FormalTy = Callee->getFunctionType()->getParamType(ArgNo);
+    Type *ActualTy = U.get()->getType();
+    if (FormalTy != ActualTy) {
+      auto *Cast = CastInst::Create(Instruction::BitCast, U.get(), FormalTy, "",
+                                    CS.getInstruction());
+      CS.setArgument(ArgNo, Cast);
+      if (Casts)
+        Casts->push_back(Cast);
+    }
+  }
+
+  // If the return type of the call site doesn't match that of the callee, cast
+  // the returned value to the appropriate type.
+  if (!CallSiteRetTy->isVoidTy() && CallSiteRetTy != CalleeRetTy)
+    createRetBitCast(CS, CallSiteRetTy, Casts);
+}
+
+void llvm::demoteCall(CallSite CS, Value *CalledValue,
+                      ArrayRef<CastInst *> Casts) {
+  assert(CS.getCalledFunction() && "Only direct call sites can be demoted");
+
+  // For each cast instruction, replace all of its uses with its source
+  // operand.
+  for (CastInst *Cast : Casts) {
+    while (!Cast->user_empty())
+      Cast->user_back()->replaceUsesOfWith(Cast, Cast->getOperand(0));
+    Cast->eraseFromParent();
+  }
+
+  // Set the called value of the call site, and mutate its function type.
+  CS.setCalledFunction(CalledValue);
+  CS.mutateFunctionType(cast<FunctionType>(
+      cast<PointerType>(CalledValue->getType())->getElementType()));
+}
+
+Instruction *llvm::promoteCallWithIfThenElse(CallSite CS, Function *Callee,
+                                             MDNode *BranchWeights) {
+
+  // Version the indirect call site. If the called value is equal to the given
+  // callee, the original call site will be executed, otherwise 'NewInst' will
+  // be executed.
+  Instruction *NewInst = versionCallSite(CS, Callee, BranchWeights);
+
+  // Promote the original call site so that it directly calls the desired
+  // function.
+  promoteCall(CS, Callee);
+
+  // Return the new indirect call.
+  return NewInst;
+}
+
+#undef DEBUG_TYPE

test/Transforms/PGOProfile/icp_covariant_call_return.ll

entry:
   %call = call noalias i8* @_Znwm(i64 8)
   %tmp = bitcast i8* %call to %struct.D*
   call void @_ZN1DC2Ev(%struct.D* %tmp)
   %tmp1 = bitcast %struct.D* %tmp to %struct.B*
   %tmp2 = bitcast %struct.B* %tmp1 to %struct.Base* (%struct.B*)***
   %vtable = load %struct.Base* (%struct.B*)**, %struct.Base* (%struct.B*)*** %tmp2, align 8
   %vfn = getelementptr inbounds %struct.Base* (%struct.B*)*, %struct.Base* (%struct.B*)** %vtable, i64 0
   %tmp3 = load %struct.Base* (%struct.B*)*, %struct.Base* (%struct.B*)** %vfn, align 8
-; ICALL-PROM:  [[BITCAST:%[0-9]+]] = bitcast %struct.Base* (%struct.B*)* %tmp3 to i8*
-; ICALL-PROM:  [[CMP:%[0-9]+]] = icmp eq i8* [[BITCAST]], bitcast (%struct.Derived* (%struct.D*)* @_ZN1D4funcEv to i8*)
+; ICALL-PROM:  [[CMP:%[0-9]+]] = icmp eq %struct.Base* (%struct.B*)* %tmp3, bitcast (%struct.Derived* (%struct.D*)* @_ZN1D4funcEv to %struct.Base* (%struct.B*)*)
 ; ICALL-PROM:  br i1 [[CMP]], label %if.true.direct_targ, label %if.false.orig_indirect, !prof [[BRANCH_WEIGHT:![0-9]+]]
 ; ICALL-PROM:if.true.direct_targ:
 ; ICALL-PROM:  [[ARG_BITCAST:%[0-9]+]] = bitcast %struct.B* %tmp1 to %struct.D*
-; ICALL-PROM:  [[DIRCALL_RET:%[0-9]+]] = call %struct.Derived* @_ZN1D4funcEv(%struct.D* [[ARG_BITCAST]])
+; ICALL-PROM:  [[DIRCALL_RET:%.+]] = call %struct.Derived* @_ZN1D4funcEv(%struct.D* [[ARG_BITCAST]])
 ; ICALL-PROM:  [[DIRCALL_RET_CAST:%[0-9]+]] = bitcast %struct.Derived* [[DIRCALL_RET]] to %struct.Base*
 ; ICALL-PROM:  br label %if.end.icp 
 ; ICALL-PROM:if.false.orig_indirect:
-; ICALL-PROM:  %call1 = call %struct.Base* %tmp3(%struct.B* %tmp1)
+; ICALL-PROM:  [[CALL1:%.+]] = call %struct.Base* %tmp3(%struct.B* %tmp1)
 ; ICALL-PROM:  br label %if.end.icp
 ; ICALL-PROM:if.end.icp:
-; ICALL-PROM:  [[PHI_RET:%[0-9]+]] = phi %struct.Base* [ %call1, %if.false.orig_indirect ], [ [[DIRCALL_RET_CAST]], %if.true.direct_targ ]
+; ICALL-PROM:  [[PHI_RET:%[0-9]+]] = phi %struct.Base* [ [[CALL1]], %if.false.orig_indirect ], [ [[DIRCALL_RET_CAST]], %if.true.direct_targ ]
   %call1 = call %struct.Base* %tmp3(%struct.B* %tmp1), !prof !1
   ret %struct.Base* %call1
 }
 
 !1 = !{!"VP", i32 0, i64 12345, i64 -3913987384944532146, i64 12345}
 ; ICALL-PROM-NOT: !1 = !{!"VP", i32 0, i64 12345, i64 -3913987384944532146, i64 12345}
 ; ICALL-PROM: [[BRANCH_WEIGHT]] = !{!"branch_weights", i32 12345, i32 0}
 ; ICALL-PROM-NOT: !1 = !{!"VP", i32 0, i64 12345, i64 -3913987384944532146, i64 12345}

test/Transforms/PGOProfile/icp_covariant_invoke_return.ll

 invoke.cont:
   %tmp = bitcast i8* %call to %struct.D*
   call void @_ZN1DC2Ev(%struct.D* %tmp)
   %tmp1 = bitcast %struct.D* %tmp to %struct.B*
   %tmp2 = bitcast %struct.B* %tmp1 to %struct.Base* (%struct.B*)***
   %vtable = load %struct.Base* (%struct.B*)**, %struct.Base* (%struct.B*)*** %tmp2, align 8
   %vfn = getelementptr inbounds %struct.Base* (%struct.B*)*, %struct.Base* (%struct.B*)** %vtable, i64 0
   %tmp3 = load %struct.Base* (%struct.B*)*, %struct.Base* (%struct.B*)** %vfn, align 8
-; ICALL-PROM:  [[BITCAST:%[0-9]+]] = bitcast %struct.Base* (%struct.B*)* %tmp3 to i8*
-; ICALL-PROM:  [[CMP:%[0-9]+]] = icmp eq i8* [[BITCAST]], bitcast (%struct.Derived* (%struct.D*)* @_ZN1D4funcEv to i8*)
+; ICALL-PROM:  [[CMP:%[0-9]+]] = icmp eq %struct.Base* (%struct.B*)* %tmp3, bitcast (%struct.Derived* (%struct.D*)* @_ZN1D4funcEv to %struct.Base* (%struct.B*)*)
 ; ICALL-PROM:  br i1 [[CMP]], label %if.true.direct_targ, label %if.false.orig_indirect, !prof [[BRANCH_WEIGHT:![0-9]+]]
 ; ICALL-PROM:if.true.direct_targ:
 ; ICALL-PROM:  [[ARG_BITCAST:%[0-9]+]] = bitcast %struct.B* %tmp1 to %struct.D*
-; ICALL-PROM:  [[DIRCALL_RET:%[0-9]+]] = invoke %struct.Derived* @_ZN1D4funcEv(%struct.D* [[ARG_BITCAST]])
-; ICALL-PROM:          to label %if.end.icp unwind label %lpad
+; ICALL-PROM:  [[DIRCALL_RET:%.+]] = invoke %struct.Derived* @_ZN1D4funcEv(%struct.D* [[ARG_BITCAST]])
+; ICALL-PROM:          to label %if.true.direct_targ.if.end.icp_crit_edge unwind label %lpad
+; ICALL-PROM:if.true.direct_targ.if.end.icp_crit_edge:
+; ICALL-PROM:  [[DIRCALL_RET_CAST:%[0-9]+]] = bitcast %struct.Derived* %call2 to %struct.Base*
+; ICALL-PROM:  br label %if.end.icp
 ; ICALL-PROM:if.false.orig_indirect:
 ; ICAll-PROM:  %call2 = invoke %struct.Base* %tmp3(%struct.B* %tmp1)
 ; ICAll-PROM:          to label %invoke.cont1 unwind label %lpad
 ; ICALL-PROM:if.end.icp:
-; ICALL-PROM:  [[DIRCALL_RET_CAST:%[0-9]+]] = bitcast %struct.Derived* [[DIRCALL_RET]] to %struct.Base*
 ; ICALL-PROM:  br label %invoke.cont1
   %call2 = invoke %struct.Base* %tmp3(%struct.B* %tmp1)
           to label %invoke.cont1 unwind label %lpad, !prof !1
 
 invoke.cont1:
 ; ICAll-PROM:  [[PHI_RET:%[0-9]+]] = phi %struct.Base* [ %call2, %if.false.orig_indirect ], [ [[DIRCALL_RET_CAST]], %if.end.icp ]
 ; ICAll-PROM:  %isnull = icmp eq %struct.Base* [[PHI_RET]], null
   %isnull = icmp eq %struct.Base* %call2, null

test/Transforms/PGOProfile/icp_invoke.ll

 define internal i32 @_ZL4bar2v() !PGOFuncName !1 {
 entry:
   ret i32 100
 }
 
 define i32 @_Z3goov() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
 entry:
   %tmp = load void ()*, void ()** @foo1, align 8
-; ICP:  [[BITCAST_IC1:%[0-9]+]] = bitcast void ()* %tmp to i8*
-; ICP:  [[CMP_IC1:%[0-9]+]] = icmp eq i8* [[BITCAST_IC1]], bitcast (void ()* @_ZL4bar1v to i8*)
+; ICP:  [[CMP_IC1:%[0-9]+]] = icmp eq void ()* %tmp, @_ZL4bar1v
 ; ICP:  br i1 [[CMP_IC1]], label %[[TRUE_LABEL_IC1:.*]], label %[[FALSE_LABEL_IC1:.*]], !prof [[BRANCH_WEIGHT:![0-9]+]]
 ; ICP:[[TRUE_LABEL_IC1]]:
 ; ICP:  invoke void @_ZL4bar1v()
 ; ICP:          to label %[[DCALL_NORMAL_DEST_IC1:.*]] unwind label %lpad
 ; ICP:[[FALSE_LABEL_IC1]]:
   invoke void %tmp()
           to label %try.cont unwind label %lpad, !prof !2
 

 
 catch:
   %tmp5 = tail call i8* @__cxa_begin_catch(i8* %tmp2)
   tail call void @__cxa_end_catch()
   br label %try.cont
 
 try.cont:
   %tmp6 = load i32 ()*, i32 ()** @foo2, align 8
-; ICP:  [[BITCAST_IC2:%[0-9]+]] = bitcast i32 ()* %tmp6 to i8*
-; ICP:  [[CMP_IC2:%[0-9]+]] = icmp eq i8* [[BITCAST_IC2]], bitcast (i32 ()* @_ZL4bar2v to i8*)
+; ICP:  [[CMP_IC2:%[0-9]+]] = icmp eq i32 ()* %tmp6, @_ZL4bar2v
 ; ICP:  br i1 [[CMP_IC2]], label %[[TRUE_LABEL_IC2:.*]], label %[[FALSE_LABEL_IC2:.*]], !prof [[BRANCH_WEIGHT:![0-9]+]]
 ; ICP:[[TRUE_LABEL_IC2]]:
-; ICP:  [[RESULT_IC2:%[0-9]+]] = invoke i32 @_ZL4bar2v()
+; ICP:  [[RESULT_IC2_0:%.+]] = invoke i32 @_ZL4bar2v()
 ; ICP:          to label %[[DCALL_NORMAL_DEST_IC2:.*]] unwind label %lpad1
 ; ICP:[[FALSE_LABEL_IC2]]:
+; ICP:  [[RESULT_IC2_1:%.+]] = invoke i32 %tmp6()
+; ICP:          to label %[[DCALL_NORMAL_DEST_IC2:.*]] unwind label %lpad1
   %call = invoke i32 %tmp6()
           to label %try.cont8 unwind label %lpad1, !prof !3
 
 ; ICP:[[DCALL_NORMAL_DEST_IC2]]:
+; ICP:  [[PHI_0:%.+]] = phi i32 [ [[RESULT_IC2_1]], %[[FALSE_LABEL_IC2]] ], [ [[RESULT_IC2_0]], %[[TRUE_LABEL_IC2]] ]
 ; ICP:  br label %try.cont8
 lpad1:
   %tmp7 = landingpad { i8*, i32 }
           catch i8* bitcast (i8** @_ZTIi to i8*)
   %tmp8 = extractvalue { i8*, i32 } %tmp7, 0
   %tmp9 = extractvalue { i8*, i32 } %tmp7, 1
   %tmp10 = tail call i32 @llvm.eh.typeid.for(i8* bitcast (i8** @_ZTIi to i8*))
   %matches5 = icmp eq i32 %tmp9, %tmp10
   br i1 %matches5, label %catch6, label %eh.resume
 
 catch6:
   %tmp11 = tail call i8* @__cxa_begin_catch(i8* %tmp8)
   tail call void @__cxa_end_catch()
   br label %try.cont8
 
 try.cont8:
   %i.0 = phi i32 [ undef, %catch6 ], [ %call, %try.cont ]
-; ICP:  %i.0 = phi i32 [ undef, %catch6 ], [ %call, %[[FALSE_LABEL_IC2]] ], [ [[RESULT_IC2]], %[[DCALL_NORMAL_DEST_IC2]] ]
+; ICP:  %i.0 = phi i32 [ undef, %catch6 ], [ [[PHI_0]], %[[DCALL_NORMAL_DEST_IC2]] ]
   ret i32 %i.0
 
 eh.resume:
   %ehselector.slot.0 = phi i32 [ %tmp9, %lpad1 ], [ %tmp3, %lpad ]
   %exn.slot.0 = phi i8* [ %tmp8, %lpad1 ], [ %tmp2, %lpad ]
   %lpad.val = insertvalue { i8*, i32 } undef, i8* %exn.slot.0, 0
   %lpad.val11 = insertvalue { i8*, i32 } %lpad.val, i32 %ehselector.slot.0, 1
   resume { i8*, i32 } %lpad.val11

test/Transforms/PGOProfile/icp_invoke_nouse.ll

 ; Function Attrs: uwtable
 define i32 @_Z3fooi(i32 %x) local_unnamed_addr personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
 entry:
   %tobool = icmp eq i32 %x, 0
   br i1 %tobool, label %if.end, label %cleanup
 
 if.end:                                           ; preds = %entry
   %fptr = load i32 ()*, i32 ()** @pfptr, align 8
-; ICP:  [[BITCAST_IC1:%[0-9]+]] = bitcast i32 ()* %fptr to i8*
-; ICP:  [[CMP_IC1:%[0-9]+]] = icmp eq i8* [[BITCAST_IC1]], bitcast (i32 ()* @_ZL4bar1v to i8*)
+; ICP:  [[CMP_IC1:%[0-9]+]] = icmp eq i32 ()* %fptr, @_ZL4bar1v
 ; ICP:  br i1 [[CMP_IC1]], label %[[TRUE_LABEL_IC1:.*]], label %[[FALSE_LABEL_IC1:.*]], !prof [[BRANCH_WEIGHT:![0-9]+]]
 ; ICP:[[TRUE_LABEL_IC1]]:
 ; ICP:  invoke i32 @_ZL4bar1v()
 ; ICP:          to label %[[DCALL_NORMAL_DEST_IC1:.*]] unwind label %lpad
 ; ICP:[[FALSE_LABEL_IC1]]:
   %call = invoke i32 %fptr()
           to label %cleanup unwind label %lpad, !prof !1
 

test/Transforms/PGOProfile/icp_mismatch_msg.ll

 ; RUN: opt < %s -pgo-icall-prom -pass-remarks-missed=pgo-icall-prom -S 2>& 1 | FileCheck %s
 ; RUN: opt < %s -passes=pgo-icall-prom -pass-remarks-missed=pgo-icall-prom -S 2>& 1 | FileCheck %s
 
-; CHECK: remark: <unknown>:0:0: Cannot promote indirect call to func4 with count of 1234: The number of arguments mismatch
+; CHECK: remark: <unknown>:0:0: Cannot promote indirect call to func4 with count of 1234: Argument number mismatch
 ; CHECK: remark: <unknown>:0:0: Cannot promote indirect call: target not found
 ; CHECK: remark: <unknown>:0:0: Cannot promote indirect call to func2 with count of 7890: Return type mismatch
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 @foo = common global i32 ()* null, align 8
 @foo2 = common global i32 ()* null, align 8

test/Transforms/PGOProfile/icp_vararg.ll

 ; RUN: opt < %s -pgo-icall-prom -S | FileCheck %s --check-prefix=ICALL-PROM
 ; RUN: opt < %s -passes=pgo-icall-prom -S | FileCheck %s --check-prefix=ICALL-PROM
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 @foo = common global i32 (i32, ...)* null, align 8
 
 define i32 @va_func(i32 %num, ...) {
 entry:
   ret i32 0
 }
 
 define i32 @bar() #1 {
 entry:
   %tmp = load i32 (i32, ...)*, i32 (i32, ...)** @foo, align 8
-; ICALL-PROM:  [[BITCAST:%[0-9]+]] = bitcast i32 (i32, ...)* %tmp to i8*
-; ICALL-PROM:  [[CMP:%[0-9]+]] = icmp eq i8* [[BITCAST]], bitcast (i32 (i32, ...)* @va_func to i8*)
+; ICALL-PROM:  [[CMP:%[0-9]+]] = icmp eq i32 (i32, ...)* %tmp, @va_func
 ; ICALL-PROM:  br i1 [[CMP]], label %if.true.direct_targ, label %if.false.orig_indirect, !prof [[BRANCH_WEIGHT:![0-9]+]]
 ; ICALL-PROM:if.true.direct_targ:
-; ICALL-PROM:  [[DIRCALL_RET:%[0-9]+]] = call i32 (i32, ...) @va_func(i32 3, i32 12, i32 22, i32 4)
+; ICALL-PROM:  [[DIRCALL_RET:%.+]] = call i32 (i32, ...) @va_func(i32 3, i32 12, i32 22, i32 4)
 ; ICALL-PROM:  br label %if.end.icp
   %call = call i32 (i32, ...) %tmp(i32 3, i32 12, i32 22, i32 4), !prof !1
 ; ICALL-PROM:if.false.orig_indirect:
-; ICALL-PROM:  %call = call i32 (i32, ...) %tmp(i32 3, i32 12, i32 22, i32 4)
+; ICALL-PROM:  [[CALL:%.+]] = call i32 (i32, ...) %tmp(i32 3, i32 12, i32 22, i32 4)
 ; ICALL-PROM:  br label %if.end.icp
   ret i32 %call
 ; ICALL-PROM:if.end.icp:
-; ICALL-PROM:  [[PHI_RET:%[0-9]+]] = phi i32 [ %call, %if.false.orig_indirect ], [ [[DIRCALL_RET]], %if.true.direct_targ ]
+; ICALL-PROM:  [[PHI_RET:%[0-9]+]] = phi i32 [ [[CALL]], %if.false.orig_indirect ], [ [[DIRCALL_RET]], %if.true.direct_targ ]
 ; ICALL-PROM:  ret i32 [[PHI_RET]] 
 
 }
 
 !1 = !{!"VP", i32 0, i64 12345, i64 989055279648259519, i64 12345}
 ; ICALL-PROM: [[BRANCH_WEIGHT]] = !{!"branch_weights", i32 12345, i32 0}

test/Transforms/PGOProfile/indirect_call_promotion.ll

 define i32 @func4() {
 entry:
   ret i32 3
 }
 
 define i32 @bar() {
 entry:
   %tmp = load i32 ()*, i32 ()** @foo, align 8
-; ICALL-PROM:   [[BITCAST:%[0-9]+]] = bitcast i32 ()* %tmp to i8*
-; ICALL-PROM:   [[CMP:%[0-9]+]] = icmp eq i8* [[BITCAST]], bitcast (i32 ()* @func4 to i8*)
+; ICALL-PROM:   [[CMP:%[0-9]+]] = icmp eq i32 ()* %tmp, @func4
 ; ICALL-PROM:   br i1 [[CMP]], label %if.true.direct_targ, label %if.false.orig_indirect, !prof [[BRANCH_WEIGHT:![0-9]+]]
 ; ICALL-PROM: if.true.direct_targ:
-; ICALL-PROM:   [[DIRCALL_RET:%[0-9]+]] = call i32 @func4()
+; ICALL-PROM:   [[DIRCALL_RET:%.+]] = call i32 @func4()
 ; ICALL-PROM-SAMPLEPGO: call i32 @func4(), !prof [[CALL_METADATA:![0-9]+]]
 ; ICALL-PROM:   br label %if.end.icp
   %call = call i32 %tmp(), !prof !1
 ; ICALL-PROM: if.false.orig_indirect:
-; ICALL-PROM:   %call = call i32 %tmp(), !prof [[NEW_VP_METADATA:![0-9]+]]
+; ICALL-PROM:   [[CALL:%.+]] = call i32 %tmp(), !prof [[NEW_VP_METADATA:![0-9]+]]
   ret i32 %call
 ; ICALL-PROM: if.end.icp:
-; ICALL-PROM:   [[PHI_RET:%[0-9]+]] = phi i32 [ %call, %if.false.orig_indirect ], [ [[DIRCALL_RET]], %if.true.direct_targ ]
+; ICALL-PROM:   [[PHI_RET:%[0-9]+]] = phi i32 [ [[CALL]], %if.false.orig_indirect ], [ [[DIRCALL_RET]], %if.true.direct_targ ]
 ; ICALL-PROM:   ret i32 [[PHI_RET]]
 }
 
 !1 = !{!"VP", i32 0, i64 1600, i64 7651369219802541373, i64 1030, i64 -4377547752858689819, i64 410, i64 -6929281286627296573, i64 150, i64 -2545542355363006406, i64 10}
 
 ; ICALL-PROM: [[BRANCH_WEIGHT]] = !{!"branch_weights", i32 1030, i32 570}
 ; ICALL-PROM: [[NEW_VP_METADATA]] = !{!"VP", i32 0, i64 570, i64 -4377547752858689819, i64 410}
 ; ICALL-PROM-SAMPLEPGO: [[CALL_METADATA]] = !{!"branch_weights", i32 1030}