[PowerPC] Add handling for ColdCC calling convention and a pass to mark candidates with coldcc attribute

[syzaara]

Summary

This patch adds support for the coldcc calling convention for Power. This changes the set of non-volatile registers. It includes a pass to stress test the implementation by marking all static directly called functions with the coldcc attribute through the option -enable-coldcc-stress-test. It also includes an option, -ppc-enable-coldcc, to add the coldcc attribute to functions which are cold at all call sites based on BlockFrequencyInfo.

[hfinkel]

lib/Target/PowerPC/PPCColdCC.cpp

1 ↗

(On Diff #117174)

Can you please merge this logic into GlobalOpt? That's where we would currently set the FastCC calling convention. We could make that logic set the FastCC, or ColdCC, calling convention as appropriate. AFAIKT, there's nothing really target-dependent here.

If need be, we can add a TTI hook to enable it for particular targets. It's not clear to me that should be necessary.

90 ↗

(On Diff #117174)

Please make this value a cl::opt so that we can easily experiment with tuning it as necessary.

120 ↗

(On Diff #117174)

Is this check actually needed? Doesn't the backend handle this like it handles FastCC (i.e., if the target does nothing special, it just falls back to using the default calling convention). If something in this patch breaks that fallback, we need to clean that up. It should always be legal to add coldcc to an internal call regardless of whether the target does anything special with it.

124 ↗

(On Diff #117174)

Reverse the order:

if (EnableColdCCStressTest || hasColdCallSite(F)) {

the cheaper-to-evaluate expression should be first.

[nemanjai]

An important aspect of the testing here is missing. Namely checking for the coldcc calling convention on the IR. You should be able to accomplish this either through an MIR test case or just a test case running opt.

lib/Target/PowerPC/PPCColdCC.cpp

1 ↗

(On Diff #117174)

I am all for making this target independent, but I would definitely opt for a TTI hook. The reasons being the following:

• A target that doesn't want to use this shouldn't be forced to incur the cost of this analysis (however small).
• Much less importantly, it is possible that some targets have incomplete/incorrect handling of this currently seldom-used calling convention. I believe a TTI hook would give people an opportunity to test this out and fix any issues they see on their targets.

9 ↗

(On Diff #117174)

Really? Not even something minimal like:

/// Sets the calling convention on cold internal functions
/// and their callsites to coldcc.

:)

27 ↗

(On Diff #117174)

I don't think it's a particularly good idea to have the DEBUG_TYPE be exactly the name of the calling convention.

50 ↗

(On Diff #117174)

Do we not want to append
ModulePass::getAnalysisUsage(AU);
here?

61 ↗

(On Diff #117174)

I'm not a fan of data members such as this. This isn't really a property of the module but the function. Furthermore, this is a data member that is only used in one function - and it's reset for every function encountered. Why is it not local to the function then?

68 ↗

(On Diff #117174)

What happens if no users are found? I'm not sure if that's possible or if it would already be discarded. But if it might still happen, seems like you don't want to bother marking it coldcc.

81 ↗

(On Diff #117174)

I don't really understand this approach. You loop over all the call sites, count how many you've seen. For each, you check if it's cold and increment the number of cold call sites. Finally, you return whether or not the two numbers are equal. Aside from the fact that the return statement is really just return NumCallSites == NumColdCallSites;, I don't understand the need to loop over all the call sites. Why is this function not just hasNonColdSite() which will return whether any sites are not cold (i.e. early exit with true if a non-cold site is found).
That seems more natural to me, but even if you want to keep this a positive test, it should:

• Be renamed to something like allCallSitesCold()
• Exit early if a non-cold site is found

87 ↗

(On Diff #117174)

This applies to all the member functions here - they're all in need of comments explaining what they do. For this one, it appears that it checks whether the block that contains the call has a BlockFrequency less than 10% of it's parent's entry block. I think you should make this explicit in the comment (of course, you'll not use 10% when you convert this to a cl::opt value, but the approach is still the same).

90 ↗

(On Diff #117174)

I agree. The option can take a percentage and please make that clear in the description of the option.

lib/Target/PowerPC/PPCFastISel.cpp

209

Why is this needed? I would imagine that with FastISel, we don't want to handle coldcc specially. I don't imagine handling it would add much (anything) to the compile time, but it just seems fundamentally like something we don't want to do - apply these ABI breaking optimizations when emitting code for -O0.

lib/Target/PowerPC/PPCFrameLowering.cpp

1953–1960

The code no longer matches the comment. You should update the comment to explain why a callee-saved register might not be live-in to the function.

1954

Variables start with uppercase.

1990–1993

This really needs a comment. It should be clear why we keep a register live after spilling if it wasn't live-in to the function.

lib/Target/PowerPC/PPCISelLowering.cpp

6392–6395

I think a ternary operator would make it more clear that you're calling the exact same function with a different argument. Same below of course.

lib/Target/PowerPC/PPCRegisterInfo.cpp

126

Why did this need to move up? It does not appear like a change that is needed since the first use of it is still below where it was before.

[syzaara]

The coldcc handling is now moved to GlobalOpt. A target can enable coldcc for internal functions which are cold at all call sites by overriding enableColdCC, or enable stress testing to mark all internal functions with coldcc by setting option -enable-coldcc-stress-test. The coldness threshold can be controlled with the option -coldcc-rel-freq.

[syzaara]

lib/Target/PowerPC/PPCFastISel.cpp

209

This is just for suppressing compiler warning messages that arise from including "PPCGenCallingConv.inc" and not having a use for the function RetCC_PPC_Cold defined in PPCGenCallingConv.inc. It's mentioned in the comment above as well.

[hfinkel]

include/llvm/Analysis/TargetTransformInfo.h

546

How about useCodeCCForColdCalls()? (I don't like enableColdCC() because coldcc is always "enabled", the question here is whether we actively form such calls.)

lib/Transforms/IPO/GlobalOpt.cpp

2216

The fastcc conversion is currently guarded by isProfitableToMakeFastCC, which really just does this:

/// Return true if this is a calling convention that we'd like to change.  The
/// idea here is that we don't want to mess with the convention if the user
/// explicitly requested something with performance implications like coldcc,
/// GHC, or anyregcc.
static bool isProfitableToMakeFastCC(Function *F) {
  CallingConv::ID CC = F->getCallingConv();
  // FIXME: Is it worth transforming x86_stdcallcc and x86_fastcallcc?
  return CC == CallingConv::C || CC == CallingConv::X86_ThisCall;
}

and we should have the same guard (which will also make sure we don't explicitly change something marked fastcc to coldcc). I recommend renaming this function to something like:

hasChangeableCC(Function *F)

and then using it in both places.

[nemanjai]

include/llvm/Analysis/TargetTransformInfo.h

546

I think that is meant to be useColdCCForColdCalls() and if so, I agree with that name.

But more importantly than the name, I don't feel that this should be an "always-on/always-off" type of query.
I feel that the target should be the arbiter of whether a specific cold callsite would provide any benefit if it were marked coldcc. Deciding that a callsite is cold is fine in GlobalOpt, but simply having a cold call in a function does not mean that marking that call with coldcc is useful.
Consider a trivial example where the caller has calls on the cold path as well as the hot path. Furthermore, all the values live across one are also live across the other. In such a case, marking the cold call as coldcc provides absolutely no gain and if the cold function is called frequently enough (i.e. either the block frequency is incorrect or the cold threshold is too high), this is a net performance loss.

So what I'm getting at is that I don't think the call frequency is enough to make the decision and I am suggesting that since the real meaning of coldcc is kind of target-specific, maybe the decision to mark functions with it should be as well. Perhaps the right thing to do would be to have the default semantics implemented in the TTI base class, allowing the target to override it.

lib/Target/PowerPC/PPCCallingConv.td

48

Period. Complete sentences in comments please.

302

The comment doesn't explain the TOC and stack pointer registers that are clearly missing from the list here. Please add a comment that we never consider the stack pointer a CSR and that we have versions of these lists with and without the TOC pointer (used depending on whether the function uses the TOC and is a non-leaf).

lib/Target/PowerPC/PPCFrameLowering.cpp

1956

Much like the comment below, this states the what but not the why. Namely, why should the registers that are live-in to the function not be marked as live-in to the block where the prologue is being inserted?

Furthermore in the below case, I can make sense of why we don't want to set the kill flag on the spill since the use will almost always come before the re-definition. But in this case, I don't really understand why we're only setting callee-saved-registers as live-in to the prologue block if they're not live-in to the function.

1992

The comment is still not adequate. All it says is what's obvious from the code - I can tell that we don't want to set the kill flag if the register is live in because the kill flag is set to !IsLiveIn. Don't state what the code does when it's so obvious, but why it does it.

lib/Target/PowerPC/PPCRegisterInfo.cpp

127

So this is now just two new empty lines?

lib/Transforms/IPO/GlobalOpt.cpp

104

It seems to me that 10% is way too high of a default. I would have expected that a call would be considered cold if it has a frequency in the range of what __builtin_expect(..., false) would produce (which I think is 1:2000). Maybe that is excessively low, but I'm fairly certain that 10% is too high.
A function being marked coldcc will end up spilling every register it uses, even if it's a leaf function. If one in 10 passes through the caller result in a call to such a "cold" function, it is likely that any improvements made to the caller as a result of the calee being coldcc are more than offset by the extra frame setup in the callee.

In any case whatever the default, I think that relying solely on the frequency won't yield good results (see my comment on the TTI hook).

2152

No braces on blocks with a single statement.

2159

changeCallSitesToColdCC()?

[syzaara]

Aside from just using the option -ppc-enable-coldcc to use coldcc on functions which are cold at all call sites, PowerPC will also check if the coldcc function is the only function called in it's callers.

[nemanjai]

Other than the tiny issue with an unnecessary cast, LGTM.

I am going to approve this since I'm holding up approval. I think @hfinkel should also have another look at this before you commit so that he can verify that you adequately addressed his comment.

lib/Target/PowerPC/PPCTargetTransformInfo.cpp

245

Hold on, the type of I is Instruction& and we're casting it to Instruction. Why?

[syzaara]

@hfinkel Can you please take another look at this before I commit?

[hfinkel]

include/llvm/Analysis/TargetTransformInfo.h

544

callsites -> call sites

545

conv. -> convention.

lib/Target/PowerPC/PPCCallingConv.td

297

Period after non-volatile.

lib/Target/PowerPC/PPCFrameLowering.cpp

1955

Can you explain here how this situation comes up? Is this because we have callee-saved registers that are callee-saved iff they're not used for parameter passing?

lib/Target/PowerPC/PPCTargetTransformInfo.cpp

231

I don't think that this implementation makes sense. A function can call several potential cold calls (which seems prohibited here), and in addition, doing the check this way seems likely to make the whole process O(N^2). Just return true here (unless there's some actual target-specific logic we'd like to have).

lib/Transforms/IPO/GlobalOpt.cpp

2124

callsite -> call site

Either you're referring to CallSite, which is a capitalized class name, or write call site (because the term is actually two words).

2136

callsites -> call sites

[syzaara]

lib/Target/PowerPC/PPCFrameLowering.cpp

1955

When the reg is used for parameter passing, it will still part of the callee-saved list. However, we won't mark it as killed when we are storing to the stack.

[hfinkel]

include/llvm/Analysis/TargetTransformInfo.h

546

Given that:

bool useColdCCForColdCalls(Function &F) const;

makes a per-function decision, call in the name shouldn't be plural. Name it useColdCCForColdCall instead.

test/Transforms/GlobalOpt/coldcc_coldsites.ll

2

This test depends on PPC TTI, and so it has to be enabled only when the backend is compiled in. You'll need to create a PowerPC subdirectory of test/Transforms/GlobalOpt and put the test in that subdirectory. The directory also needs a lit.local.cfg. Copy the ones from test/Transforms/LoopVectorize/PowerPC/lit.local.cfg

[hfinkel]

lib/Transforms/IPO/GlobalOpt.cpp

2138

returning false if the caller function contains other non cold calls.

I still don't understand the purpose of this check. The rationale needs to be clearly explained (or just get rid of it). If a function is cold at all call sites, why not convert it to use coldcc? Why does it matter if any of its callers also have non-cold calls?

[syzaara]

lib/Transforms/IPO/GlobalOpt.cpp

2138

The motivation to use coldcc is to reduce the caller saved register spills in the caller, however if there is another non coldcc call site, the caller will have to spill the caller saved registers anyway. This defeats the purpose of using coldcc and having the overhead to spill all registers in the cold callee and then again spilling the caller saved registers in the caller around the non coldcc call site. This is why we want to only use coldcc for a call site in a function if all the calls sites in the function can use coldcc.

[hfinkel]

lib/Transforms/IPO/GlobalOpt.cpp

2138

If I'm misunderstanding what's going on, please correct me, but as I understand this: If I have a bunch of functions, A, B, C, and D, and these all have a cold path calling F (and no other function calls), then F is converted to be coldcc and all functions benefit from this optimization. Then, one day, an unsuspecting programmer adds a function call into some non-cold path in functions A. Now, suddenly, the performance of B, C, and D is also worse (because we're now no longer converting F to be coldcc, even though it is still only called on cold paths in all functions). This seems Bad. Obviously, sometimes these kinds of non-local effects are unavoidable, but when they're avoidable, we should avoid them. They're horrible for usability. The only justification for doing this would be if, by continuing to convert F to be coldcc, the performance of A would now be worse, but it doesn't seem like that should be the case (and it still might be better, depending on how we do shrink wrapping and where the other call is).

include/llvm/Analysis/TargetTransformInfo.h

public:
   /// \brief Return true if switches should be turned into lookup tables for the
   /// target.
   bool shouldBuildLookupTables() const;
 
   /// \brief Return true if switches should be turned into lookup tables
   /// containing this constant value for the target.
   bool shouldBuildLookupTablesForConstant(Constant *C) const;
 
+  /// \brief Return true if the input function which is cold at all callsites,
+  ///  should use coldcc calling conv.
+  bool useColdCCForColdCalls(Function &F) const;
+
   unsigned getScalarizationOverhead(Type *Ty, bool Insert, bool Extract) const;
 
   unsigned getOperandsScalarizationOverhead(ArrayRef<const Value *> Args,
                                             unsigned VF) const;
 
   /// If target has efficient vector element load/store instructions, it can
   /// return true here so that insertion/extraction costs are not added to
   /// the scalarization cost of a load/store.
public:
   virtual bool LSRWithInstrQueries() = 0;
   virtual bool isTruncateFree(Type *Ty1, Type *Ty2) = 0;
   virtual bool isProfitableToHoist(Instruction *I) = 0;
   virtual bool isTypeLegal(Type *Ty) = 0;
   virtual unsigned getJumpBufAlignment() = 0;
   virtual unsigned getJumpBufSize() = 0;
   virtual bool shouldBuildLookupTables() = 0;
   virtual bool shouldBuildLookupTablesForConstant(Constant *C) = 0;
+  virtual bool useColdCCForColdCalls(Function &F) = 0;
   virtual unsigned
   getScalarizationOverhead(Type *Ty, bool Insert, bool Extract) = 0;
   virtual unsigned getOperandsScalarizationOverhead(ArrayRef<const Value *> Args,
                                                     unsigned VF) = 0;
   virtual bool supportsEfficientVectorElementLoadStore() = 0;
   virtual bool enableAggressiveInterleaving(bool LoopHasReductions) = 0;
   virtual const MemCmpExpansionOptions *enableMemCmpExpansion(
       bool IsZeroCmp) const = 0;
public:
   unsigned getJumpBufAlignment() override { return Impl.getJumpBufAlignment(); }
   unsigned getJumpBufSize() override { return Impl.getJumpBufSize(); }
   bool shouldBuildLookupTables() override {
     return Impl.shouldBuildLookupTables();
   }
   bool shouldBuildLookupTablesForConstant(Constant *C) override {
     return Impl.shouldBuildLookupTablesForConstant(C);
   }
+  bool useColdCCForColdCalls(Function &F) override {
+    return Impl.useColdCCForColdCalls(F);
+  }
+
   unsigned getScalarizationOverhead(Type *Ty, bool Insert,
                                     bool Extract) override {
     return Impl.getScalarizationOverhead(Ty, Insert, Extract);
   }
   unsigned getOperandsScalarizationOverhead(ArrayRef<const Value *> Args,
                                             unsigned VF) override {
     return Impl.getOperandsScalarizationOverhead(Args, VF);
   }

include/llvm/Analysis/TargetTransformInfoImpl.h

public:
 
   unsigned getJumpBufAlignment() { return 0; }
 
   unsigned getJumpBufSize() { return 0; }
 
   bool shouldBuildLookupTables() { return true; }
   bool shouldBuildLookupTablesForConstant(Constant *C) { return true; }
 
+  bool useColdCCForColdCalls(Function &F) { return false; }
+
   unsigned getScalarizationOverhead(Type *Ty, bool Insert, bool Extract) {
     return 0;
   }
 
   unsigned getOperandsScalarizationOverhead(ArrayRef<const Value *> Args,
                                             unsigned VF) { return 0; }
 
   bool supportsEfficientVectorElementLoadStore() { return false; }

lib/Analysis/TargetTransformInfo.cpp

 
 bool TargetTransformInfo::shouldBuildLookupTables() const {
   return TTIImpl->shouldBuildLookupTables();
 }
 bool TargetTransformInfo::shouldBuildLookupTablesForConstant(Constant *C) const {
   return TTIImpl->shouldBuildLookupTablesForConstant(C);
 }
 
+bool TargetTransformInfo::useColdCCForColdCalls(Function &F) const {
+  return TTIImpl->useColdCCForColdCalls(F);
+}
+
 unsigned TargetTransformInfo::
 getScalarizationOverhead(Type *Ty, bool Insert, bool Extract) const {
   return TTIImpl->getScalarizationOverhead(Ty, Insert, Extract);
 }
 
 unsigned TargetTransformInfo::
 getOperandsScalarizationOverhead(ArrayRef<const Value *> Args,
                                  unsigned VF) const {

lib/Target/PowerPC/PPCCallingConv.td

 //
 // This calling convention is currently only supported by the stackmap and
 // patchpoint intrinsics. All other uses will result in an assert on Debug
 // builds. On Release builds we fallback to the PPC C calling convention.
 def RetCC_PPC64_AnyReg : CallingConv<[
   CCCustom<"CC_PPC_AnyReg_Error">
 ]>;
 
+// Return-value convention for PowerPC coldcc.
+def RetCC_PPC_Cold : CallingConv<[
+  // Use the same return registers as RetCC_PPC, but limited to only
+  // one return value. The remaining return values will be saved to
+  // the stack.
+  CCIfType<[i32, i1], CCIfSubtarget<"isPPC64()", CCPromoteToType<i64>>>,
+  CCIfType<[i1], CCIfNotSubtarget<"isPPC64()", CCPromoteToType<i32>>>,
+
+  CCIfType<[i32], CCAssignToReg<[R3]>>,
+  CCIfType<[i64], CCAssignToReg<[X3]>>,
+  CCIfType<[i128], CCAssignToReg<[X3]>>,
+
+  CCIfType<[f32], CCAssignToReg<[F1]>>,
+  CCIfType<[f64], CCAssignToReg<[F1]>>,
+
+  CCIfType<[v4f64, v4f32, v4i1],
+           CCIfSubtarget<"hasQPX()", CCAssignToReg<[QF1]>>>,
+
+  CCIfType<[v16i8, v8i16, v4i32, v2i64, v1i128, v4f32, v2f64],
+           CCIfSubtarget<"hasAltivec()",
+           CCAssignToReg<[V2]>>>
+]>;
+
 // Return-value convention for PowerPC
 def RetCC_PPC : CallingConv<[
   CCIfCC<"CallingConv::AnyReg", CCDelegateTo<RetCC_PPC64_AnyReg>>,
 
   // On PPC64, integer return values are always promoted to i64
   CCIfType<[i32, i1], CCIfSubtarget<"isPPC64()", CCPromoteToType<i64>>>,
   CCIfType<[i1], CCIfNotSubtarget<"isPPC64()", CCPromoteToType<i32>>>,
 

 def CSR_SVR464_R2_ViaCopy : CalleeSavedRegs<(add CSR_SVR464_R2)>;
 
 def CSR_SVR464_R2_Altivec : CalleeSavedRegs<(add CSR_SVR464_Altivec, X2)>;
 
 def CSR_SVR464_R2_Altivec_ViaCopy : CalleeSavedRegs<(add CSR_SVR464_R2_Altivec)>;
 
 def CSR_NoRegs : CalleeSavedRegs<(add)>;
 
+// coldcc calling convection marks most registers as non-volatile
+// Do not include r1 since the stack pointer is never considered a CSR.
+// Do not include r2, since it is the TOC register and is added depending
+// on wether or not the function uses the TOC and is a non-leaf.
+// Do not include r0,r11,r13 as they are optional in functional linkage
+// and value may be altered by inter-library calls.
+// Do not include r12 as it is used as a scratch register.
+// Do not include return registers r3, f1, v2.
+def CSR_SVR32_ColdCC : CalleeSavedRegs<(add (sequence "R%u", 4, 10),
+                                          (sequence "R%u", 14, 31),
+                                          F0, (sequence "F%u", 2, 31),
+                                          (sequence "CR%u", 0, 7))>;
+
+def CSR_SVR32_ColdCC_Altivec : CalleeSavedRegs<(add CSR_SVR32_ColdCC,
+                                            (sequence "V%u", 0, 1),
+                                            (sequence "V%u", 3, 31))>;
+
+def CSR_SVR64_ColdCC : CalleeSavedRegs<(add  (sequence "X%u", 4, 10),
+                                             (sequence "X%u", 14, 31),
+                                             F0, (sequence "F%u", 2, 31),
+                                             (sequence "CR%u", 0, 7))>;
+
+def CSR_SVR64_ColdCC_R2: CalleeSavedRegs<(add CSR_SVR64_ColdCC, X2)>;
+
+def CSR_SVR64_ColdCC_Altivec : CalleeSavedRegs<(add CSR_SVR64_ColdCC,
+                                             (sequence "V%u", 0, 1),
+                                             (sequence "V%u", 3, 31))>;
+
+def CSR_SVR64_ColdCC_R2_Altivec : CalleeSavedRegs<(add CSR_SVR64_ColdCC_Altivec, X2)>;
+
 def CSR_64_AllRegs: CalleeSavedRegs<(add X0, (sequence "X%u", 3, 10),
                                              (sequence "X%u", 14, 31),
                                              (sequence "F%u", 0, 31),
                                              (sequence "CR%u", 0, 7))>;
 
 def CSR_64_AllRegs_Altivec : CalleeSavedRegs<(add CSR_64_AllRegs,
                                              (sequence "V%u", 0, 31))>;
 
 def CSR_64_AllRegs_VSX : CalleeSavedRegs<(add CSR_64_AllRegs_Altivec,
                                          (sequence "VSL%u", 0, 31))>;
 

lib/Target/PowerPC/PPCFastISel.cpp

 // stemming from unused functions included from PPCGenCallingConv.inc.
 CCAssignFn *PPCFastISel::usePPC32CCs(unsigned Flag) {
   if (Flag == 1)
     return CC_PPC32_SVR4;
   else if (Flag == 2)
     return CC_PPC32_SVR4_ByVal;
   else if (Flag == 3)
     return CC_PPC32_SVR4_VarArg;
+  else if (Flag == 4)
+    return RetCC_PPC_Cold;
   else
     return RetCC_PPC;
 }
 
 static Optional<PPC::Predicate> getComparePred(CmpInst::Predicate Pred) {
   switch (Pred) {
     // These are not representable with any single compare.
     case CmpInst::FCMP_FALSE:

lib/Target/PowerPC/PPCFrameLowering.cpp

for (unsigned i = 0, e = CSI.size(); i != e; ++i) {
     // Darwin, ignore it.
     if (Reg == PPC::VRSAVE && !Subtarget.isDarwinABI())
       continue;
 
     // CR2 through CR4 are the nonvolatile CR fields.
     bool IsCRField = PPC::CR2 <= Reg && Reg <= PPC::CR4;
 
     // Add the callee-saved register as live-in; it's killed at the spill.
+    // Do not do this for callee-saved registers that are live-in to the
+    // function because they will already be marked live-in and this will be
+    // adding it for a second time. It is an error to add the same register
+    // to the set more than once.
+    const MachineRegisterInfo &MRI = MF->getRegInfo();
+    bool IsLiveIn = MRI.isLiveIn(Reg);
+    if (!IsLiveIn)
        MBB.addLiveIn(Reg);
 
     if (CRSpilled && IsCRField) {
       CRMIB.addReg(Reg, RegState::ImplicitKill);
       continue;
     }
 
     // Insert the spill to the stack frame.
     if (IsCRField) {
if (IsCRField) {
         MBB.insert(MI, CRMIB);
         MBB.insert(MI, addFrameReference(BuildMI(*MF, DL, TII.get(PPC::STW))
                                          .addReg(PPC::R12,
                                                  getKillRegState(true)),
                                          CSI[i].getFrameIdx()));
       }
     } else {
       const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg);
-      TII.storeRegToStackSlot(MBB, MI, Reg, true,
+      // Use !IsLiveIn for the kill flag.
+      // We do not want to kill registers that are live in this function
+      // before their use because they will become undefined registers.
+      TII.storeRegToStackSlot(MBB, MI, Reg, !IsLiveIn,
                               CSI[i].getFrameIdx(), RC, TRI);
     }
   }
   return true;
 }
 
 static void
 restoreCRs(bool isPPC64, bool is31,

lib/Target/PowerPC/PPCISelLowering.cpp

 
 SDValue PPCTargetLowering::LowerCallResult(
     SDValue Chain, SDValue InFlag, CallingConv::ID CallConv, bool isVarArg,
     const SmallVectorImpl<ISD::InputArg> &Ins, const SDLoc &dl,
     SelectionDAG &DAG, SmallVectorImpl<SDValue> &InVals) const {
   SmallVector<CCValAssign, 16> RVLocs;
   CCState CCRetInfo(CallConv, isVarArg, DAG.getMachineFunction(), RVLocs,
                     *DAG.getContext());
-  CCRetInfo.AnalyzeCallResult(Ins, RetCC_PPC);
+
+  CCRetInfo.AnalyzeCallResult(
+      Ins, (Subtarget.isSVR4ABI() && CallConv == CallingConv::Cold)
+               ? RetCC_PPC_Cold
+               : RetCC_PPC);
 
   // Copy all of the result registers out of their specified physreg.
   for (unsigned i = 0, e = RVLocs.size(); i != e; ++i) {
     CCValAssign &VA = RVLocs[i];
     assert(VA.isRegLoc() && "Can only return in registers!");
 
     SDValue Val = DAG.getCopyFromReg(Chain, dl,
                                      VA.getLocReg(), VA.getLocVT(), InFlag);
SDValue PPCTargetLowering::LowerCall_32SVR4(
     const SmallVectorImpl<SDValue> &OutVals,
     const SmallVectorImpl<ISD::InputArg> &Ins, const SDLoc &dl,
     SelectionDAG &DAG, SmallVectorImpl<SDValue> &InVals,
     ImmutableCallSite CS) const {
   // See PPCTargetLowering::LowerFormalArguments_32SVR4() for a description
   // of the 32-bit SVR4 ABI stack frame layout.
 
   assert((CallConv == CallingConv::C ||
+          CallConv == CallingConv::Cold ||
           CallConv == CallingConv::Fast) && "Unknown calling convention!");
 
   unsigned PtrByteSize = 4;
 
   MachineFunction &MF = DAG.getMachineFunction();
 
   // Mark this function as potentially containing a function that contains a
   // tail call. As a consequence the frame pointer will be used for dynamicalloc

 
 bool
 PPCTargetLowering::CanLowerReturn(CallingConv::ID CallConv,
                                   MachineFunction &MF, bool isVarArg,
                                   const SmallVectorImpl<ISD::OutputArg> &Outs,
                                   LLVMContext &Context) const {
   SmallVector<CCValAssign, 16> RVLocs;
   CCState CCInfo(CallConv, isVarArg, MF, RVLocs, Context);
-  return CCInfo.CheckReturn(Outs, RetCC_PPC);
+  return CCInfo.CheckReturn(
+      Outs, (Subtarget.isSVR4ABI() && CallConv == CallingConv::Cold)
+                ? RetCC_PPC_Cold
+                : RetCC_PPC);
 }
 
 SDValue
 PPCTargetLowering::LowerReturn(SDValue Chain, CallingConv::ID CallConv,
                                bool isVarArg,
                                const SmallVectorImpl<ISD::OutputArg> &Outs,
                                const SmallVectorImpl<SDValue> &OutVals,
                                const SDLoc &dl, SelectionDAG &DAG) const {
   SmallVector<CCValAssign, 16> RVLocs;
   CCState CCInfo(CallConv, isVarArg, DAG.getMachineFunction(), RVLocs,
                  *DAG.getContext());
-  CCInfo.AnalyzeReturn(Outs, RetCC_PPC);
+  CCInfo.AnalyzeReturn(Outs,
+                       (Subtarget.isSVR4ABI() && CallConv == CallingConv::Cold)
+                           ? RetCC_PPC_Cold
+                           : RetCC_PPC);
 
   SDValue Flag;
   SmallVector<SDValue, 4> RetOps(1, Chain);
 
   // Copy the result values into the output registers.
   for (unsigned i = 0; i != RVLocs.size(); ++i) {
     CCValAssign &VA = RVLocs[i];
     assert(VA.isRegLoc() && "Can only return in registers!");

lib/Target/PowerPC/PPCRegisterInfo.cpp

PPCRegisterInfo::getPointerRegClass(const MachineFunction &MF, unsigned Kind)
   if (TM.isPPC64())
     return &PPC::G8RCRegClass;
   return &PPC::GPRCRegClass;
 }
 
 const MCPhysReg*
 PPCRegisterInfo::getCalleeSavedRegs(const MachineFunction *MF) const {
   const PPCSubtarget &Subtarget = MF->getSubtarget<PPCSubtarget>();
   if (MF->getFunction()->getCallingConv() == CallingConv::AnyReg) {
     if (Subtarget.hasVSX())
       return CSR_64_AllRegs_VSX_SaveList;
     if (Subtarget.hasAltivec())
       return CSR_64_AllRegs_Altivec_SaveList;
     return CSR_64_AllRegs_SaveList;
   }
 
   if (Subtarget.isDarwinABI())
     return TM.isPPC64()
                ? (Subtarget.hasAltivec() ? CSR_Darwin64_Altivec_SaveList
                                          : CSR_Darwin64_SaveList)
                : (Subtarget.hasAltivec() ? CSR_Darwin32_Altivec_SaveList
                                          : CSR_Darwin32_SaveList);
 
   if (TM.isPPC64() && MF->getInfo<PPCFunctionInfo>()->isSplitCSR())
     return CSR_SRV464_TLS_PE_SaveList;
 
   // On PPC64, we might need to save r2 (but only if it is not reserved).
   bool SaveR2 = MF->getRegInfo().isAllocatable(PPC::X2);
 
+  if (MF->getFunction()->getCallingConv() == CallingConv::Cold) {
+    return TM.isPPC64()
+               ? (Subtarget.hasAltivec()
+                      ? (SaveR2 ? CSR_SVR64_ColdCC_R2_Altivec_SaveList
+                                : CSR_SVR64_ColdCC_Altivec_SaveList)
+                      : (SaveR2 ? CSR_SVR64_ColdCC_R2_SaveList
+                                : CSR_SVR64_ColdCC_SaveList))
+               : (Subtarget.hasAltivec() ? CSR_SVR32_ColdCC_Altivec_SaveList
+                                         : CSR_SVR32_ColdCC_SaveList);
+  }
+
   return TM.isPPC64()
              ? (Subtarget.hasAltivec()
                     ? (SaveR2 ? CSR_SVR464_R2_Altivec_SaveList
                               : CSR_SVR464_Altivec_SaveList)
                     : (SaveR2 ? CSR_SVR464_R2_SaveList : CSR_SVR464_SaveList))
              : (Subtarget.hasAltivec() ? CSR_SVR432_Altivec_SaveList
                                        : CSR_SVR432_SaveList);
 }
PPCRegisterInfo::getCallPreservedMask(const MachineFunction &MF,
   }
 
   if (Subtarget.isDarwinABI())
     return TM.isPPC64() ? (Subtarget.hasAltivec() ? CSR_Darwin64_Altivec_RegMask
                                                   : CSR_Darwin64_RegMask)
                         : (Subtarget.hasAltivec() ? CSR_Darwin32_Altivec_RegMask
                                                   : CSR_Darwin32_RegMask);
 
+  if (CC == CallingConv::Cold) {
+    return TM.isPPC64() ? (Subtarget.hasAltivec() ? CSR_SVR64_ColdCC_Altivec_RegMask
+                                                  : CSR_SVR64_ColdCC_RegMask)
+                        : (Subtarget.hasAltivec() ? CSR_SVR32_ColdCC_Altivec_RegMask
+                                                  : CSR_SVR32_ColdCC_RegMask);
+  }
+
   return TM.isPPC64() ? (Subtarget.hasAltivec() ? CSR_SVR464_Altivec_RegMask
                                                 : CSR_SVR464_RegMask)
                       : (Subtarget.hasAltivec() ? CSR_SVR432_Altivec_RegMask
                                                 : CSR_SVR432_RegMask);
 }
 
 const uint32_t*
 PPCRegisterInfo::getNoPreservedMask() const {

lib/Target/PowerPC/PPCTargetTransformInfo.h

public:
   TTI::PopcntSupportKind getPopcntSupport(unsigned TyWidth);
   void getUnrollingPreferences(Loop *L, ScalarEvolution &SE,
                                TTI::UnrollingPreferences &UP);
 
   /// @}
 
   /// \name Vector TTI Implementations
   /// @{
-
+  bool useColdCCForColdCalls(Function &F);
   bool enableAggressiveInterleaving(bool LoopHasReductions);
   const TTI::MemCmpExpansionOptions *enableMemCmpExpansion(
       bool IsZeroCmp) const;
   bool enableInterleavedAccessVectorization();
   unsigned getNumberOfRegisters(bool Vector);
   unsigned getRegisterBitWidth(bool Vector) const;
   unsigned getCacheLineSize();
   unsigned getPrefetchDistance();

lib/Target/PowerPC/PPCTargetTransformInfo.cpp

 cl::desc("disable constant hoisting on PPC"), cl::init(false), cl::Hidden);
 
 // This is currently only used for the data prefetch pass which is only enabled
 // for BG/Q by default.
 static cl::opt<unsigned>
 CacheLineSize("ppc-loop-prefetch-cache-line", cl::Hidden, cl::init(64),
               cl::desc("The loop prefetch cache line size"));
 
+static cl::opt<bool>
+EnablePPCColdCC("ppc-enable-coldcc", cl::Hidden, cl::init(false),
+                cl::desc("Enable using coldcc calling conv for cold "
+                         "internal functions"));
+
 //===----------------------------------------------------------------------===//
 //
 // PPC cost model.
 //
 //===----------------------------------------------------------------------===//
 
 TargetTransformInfo::PopcntSupportKind
 PPCTTIImpl::getPopcntSupport(unsigned TyWidth) {
if (ST->getDarwinDirective() == PPC::DIR_A2) {
     // We unroll a lot on the A2 (hundreds of instructions), and the benefits
     // often outweigh the cost of a division to compute the trip count.
     UP.AllowExpensiveTripCount = true;
   }
 
   BaseT::getUnrollingPreferences(L, SE, UP);
 }
 
+// This function returns true if the input function can be marked coldcc.
+// It does this by finding all call sites of this function and checking
+// that no other functions are called aside from the coldcc function by
+// the callers.
+bool PPCTTIImpl::useColdCCForColdCalls(Function &F) {
+  if (!EnablePPCColdCC)
+    return false;
+
+  if (F.user_empty())
+    return false;
+
+  for (User *U : F.users()) {
+    if (isa<BlockAddress>(U))
+      continue;
+
+    CallSite CS(cast<Instruction>(U));
+    Function *CallerFunc = CS.getInstruction()->getParent()->getParent();
+    // Check if the caller func has any other callsites aside from a call to
+    // the input function.
+    for (BasicBlock &B : *CallerFunc) {
+      for (Instruction &I : B) {
+        if (isa<CallInst>(I)) {
+          CallSite CS2(cast<Instruction>(&I));
+          if (CS2 != CS)
+            return false;
+        }
+      }
+    }
+  }
+
+  return true;
+}
+
 bool PPCTTIImpl::enableAggressiveInterleaving(bool LoopHasReductions) {
   // On the A2, always unroll aggressively. For QPX unaligned loads, we depend
   // on combining the loads generated for consecutive accesses, and failure to
   // do so is particularly expensive. This makes it much more likely (compared
   // to only using concatenation unrolling).
   if (ST->getDarwinDirective() == PPC::DIR_A2)
     return true;
 

lib/Transforms/IPO/GlobalOpt.cpp

 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/ADT/iterator_range.h"
+#include "llvm/Analysis/BlockFrequencyInfo.h"
 #include "llvm/Analysis/ConstantFolding.h"
 #include "llvm/Analysis/MemoryBuiltins.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
+#include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/BinaryFormat/Dwarf.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CallSite.h"
 #include "llvm/IR/CallingConv.h"
 #include "llvm/IR/Constant.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DataLayout.h"

 #include "llvm/IR/Type.h"
 #include "llvm/IR/Use.h"
 #include "llvm/IR/User.h"
 #include "llvm/IR/Value.h"
 #include "llvm/IR/ValueHandle.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/AtomicOrdering.h"
 #include "llvm/Support/Casting.h"
+#include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/IPO.h"
 #include "llvm/Transforms/Utils/CtorUtils.h"
 #include "llvm/Transforms/Utils/Evaluator.h"
 #include "llvm/Transforms/Utils/GlobalStatus.h"

 STATISTIC(NumLocalized , "Number of globals localized");
 STATISTIC(NumShrunkToBool  , "Number of global vars shrunk to booleans");
 STATISTIC(NumFastCallFns   , "Number of functions converted to fastcc");
 STATISTIC(NumCtorsEvaluated, "Number of static ctors evaluated");
 STATISTIC(NumNestRemoved   , "Number of nest attributes removed");
 STATISTIC(NumAliasesResolved, "Number of global aliases resolved");
 STATISTIC(NumAliasesRemoved, "Number of global aliases eliminated");
 STATISTIC(NumCXXDtorsRemoved, "Number of global C++ destructors removed");
+STATISTIC(NumInternalFunc, "Number of internal functions");
+STATISTIC(NumColdCC, "Number of functions marked coldcc");
+
+static cl::opt<bool>
+    EnableColdCCStressTest("enable-coldcc-stress-test",
+                           cl::desc("Enable stress test of coldcc by adding "
+                                    "calling conv to all internal functions."),
+                           cl::init(false), cl::Hidden);
+
+static cl::opt<int> ColdCCRelFreq(
+    "coldcc-rel-freq", cl::Hidden, cl::init(2), cl::ZeroOrMore,
+    cl::desc(
+        "Maxmimum block frequency, expressed as a percentage of caller's "
+        "entry frequency, for a callsite to be considered cold for enabling"
+        "coldcc"));
 
 /// Is this global variable possibly used by a leak checker as a root?  If so,
 /// we might not really want to eliminate the stores to it.
 static bool isLeakCheckerRoot(GlobalVariable *GV) {
   // A global variable is a root if it is a pointer, or could plausibly contain
   // a pointer.  There are two challenges; one is that we could have a struct
   // the has an inner member which is a pointer.  We recurse through the type to
   // detect these (up to a point).  The other is that we may actually be a union
for (User *U : F->users()) {
     CS.setAttributes(StripNest(F->getContext(), CS.getAttributes()));
   }
 }
 
 /// Return true if this is a calling convention that we'd like to change.  The
 /// idea here is that we don't want to mess with the convention if the user
 /// explicitly requested something with performance implications like coldcc,
 /// GHC, or anyregcc.
-static bool isProfitableToMakeFastCC(Function *F) {
+static bool hasChangeableCC(Function *F) {
   CallingConv::ID CC = F->getCallingConv();
   // FIXME: Is it worth transforming x86_stdcallcc and x86_fastcallcc?
   return CC == CallingConv::C || CC == CallingConv::X86_ThisCall;
 }
 
+/// Return true if the block containing the callsite has a BlockFrequency of
+/// less than ColdCCRelFreq% of the entry block.
+static bool isColdCallSite(CallSite CS, BlockFrequencyInfo &CallerBFI) {
+  const BranchProbability ColdProb(ColdCCRelFreq, 100);
+  auto CallSiteBB = CS.getInstruction()->getParent();
+  auto CallSiteFreq = CallerBFI.getBlockFreq(CallSiteBB);
+  auto CallerEntryFreq =
+      CallerBFI.getBlockFreq(&(CS.getCaller()->getEntryBlock()));
+  return CallSiteFreq < CallerEntryFreq * ColdProb;
+}
+
+
+/// Return true if all blocks that contain callsites to function F are
+/// considered cold by isColdCallSite.
+static bool
+allCallSitesCold(Function &F,
+                 function_ref<BlockFrequencyInfo &(Function &)> GetBFI) {
+
+  if (F.user_empty())
+    return false;
+
+  for (User *U : F.users()) {
+    if (isa<BlockAddress>(U))
+      continue;
+
+    CallSite CS(cast<Instruction>(U));
+    Function *CallerFunc = CS.getInstruction()->getParent()->getParent();
+    BlockFrequencyInfo &CallerBFI = GetBFI(*CallerFunc);
+    if (!isColdCallSite(CS, CallerBFI))
+      return false;
+  }
+  return true;
+}
+
+static void changeCallSitesToColdCC(Function *F) {
+  for (User *U : F->users()) {
+    if (isa<BlockAddress>(U))
+      continue;
+    CallSite CS(cast<Instruction>(U));
+    CS.setCallingConv(CallingConv::Cold);
+  }
+}
+
 static bool
 OptimizeFunctions(Module &M, TargetLibraryInfo *TLI,
+                  function_ref<TargetTransformInfo &(Function &)> GetTTI,
+                  function_ref<BlockFrequencyInfo &(Function &)> GetBFI,
                   function_ref<DominatorTree &(Function &)> LookupDomTree,
                   SmallSet<const Comdat *, 8> &NotDiscardableComdats) {
+
   bool Changed = false;
   // Optimize functions.
   for (Module::iterator FI = M.begin(), E = M.end(); FI != E; ) {
     Function *F = &*FI++;
+
     // Functions without names cannot be referenced outside this module.
     if (!F->hasName() && !F->isDeclaration() && !F->hasLocalLinkage())
       F->setLinkage(GlobalValue::InternalLinkage);
 
     if (deleteIfDead(*F, NotDiscardableComdats)) {
       Changed = true;
       continue;
     }
if (!F->isDeclaration()) {
         Changed = true;
       }
     }
 
     Changed |= processGlobal(*F, TLI, LookupDomTree);
 
     if (!F->hasLocalLinkage())
       continue;
-    if (isProfitableToMakeFastCC(F) && !F->isVarArg() &&
+
+    if (hasChangeableCC(F) && !F->isVarArg() &&
+        !F->hasAddressTaken()) {
+      NumInternalFunc++;
+      TargetTransformInfo &TTI = GetTTI(*F);
+      // Change the calling convention to coldcc if either stress testing is
+      // enabled or the target would like to use coldcc on the specified
+      // function which is cold at all call sites.
+      if (EnableColdCCStressTest ||
+          (allCallSitesCold(*F, GetBFI) && TTI.useColdCCForColdCalls(*F))) {
+        F->setCallingConv(CallingConv::Cold);
+        changeCallSitesToColdCC(F);
+        Changed = true;
+        NumColdCC++;
+      }
+    }
+
+    if (hasChangeableCC(F) && !F->isVarArg() &&
         !F->hasAddressTaken()) {
       // If this function has a calling convention worth changing, is not a
       // varargs function, and is only called directly, promote it to use the
       // Fast calling convention.
       F->setCallingConv(CallingConv::Fast);
       ChangeCalleesToFastCall(F);
       ++NumFastCallFns;
       Changed = true;
for (auto I = CXAAtExitFn->user_begin(), E = CXAAtExitFn->user_end();
     Changed |= true;
   }
 
   return Changed;
 }
 
 static bool optimizeGlobalsInModule(
     Module &M, const DataLayout &DL, TargetLibraryInfo *TLI,
+    function_ref<TargetTransformInfo &(Function &)> GetTTI,
+    function_ref<BlockFrequencyInfo &(Function &)> GetBFI,
     function_ref<DominatorTree &(Function &)> LookupDomTree) {
   SmallSet<const Comdat *, 8> NotDiscardableComdats;
   bool Changed = false;
   bool LocalChange = true;
   while (LocalChange) {
     LocalChange = false;
 
     NotDiscardableComdats.clear();
     for (const GlobalVariable &GV : M.globals())
       if (const Comdat *C = GV.getComdat())
         if (!GV.isDiscardableIfUnused() || !GV.use_empty())
           NotDiscardableComdats.insert(C);
     for (Function &F : M)
       if (const Comdat *C = F.getComdat())
         if (!F.isDefTriviallyDead())
           NotDiscardableComdats.insert(C);
     for (GlobalAlias &GA : M.aliases())
       if (const Comdat *C = GA.getComdat())
         if (!GA.isDiscardableIfUnused() || !GA.use_empty())
           NotDiscardableComdats.insert(C);
 
     // Delete functions that are trivially dead, ccc -> fastcc
-    LocalChange |=
-        OptimizeFunctions(M, TLI, LookupDomTree, NotDiscardableComdats);
+    LocalChange |= OptimizeFunctions(M, TLI, GetTTI, GetBFI, LookupDomTree,
+                                     NotDiscardableComdats);
 
     // Optimize global_ctors list.
     LocalChange |= optimizeGlobalCtorsList(M, [&](Function *F) {
       return EvaluateStaticConstructor(F, DL, TLI);
     });
 
     // Optimize non-address-taken globals.
     LocalChange |= OptimizeGlobalVars(M, TLI, LookupDomTree,

 PreservedAnalyses GlobalOptPass::run(Module &M, ModuleAnalysisManager &AM) {
     auto &DL = M.getDataLayout();
     auto &TLI = AM.getResult<TargetLibraryAnalysis>(M);
     auto &FAM =
         AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
     auto LookupDomTree = [&FAM](Function &F) -> DominatorTree &{
       return FAM.getResult<DominatorTreeAnalysis>(F);
     };
-    if (!optimizeGlobalsInModule(M, DL, &TLI, LookupDomTree))
+    auto GetTTI = [&FAM](Function &F) -> TargetTransformInfo & {
+      return FAM.getResult<TargetIRAnalysis>(F);
+    };
+
+    auto GetBFI = [&FAM](Function &F) -> BlockFrequencyInfo & {
+      return FAM.getResult<BlockFrequencyAnalysis>(F);
+    };
+
+    if (!optimizeGlobalsInModule(M, DL, &TLI, GetTTI, GetBFI, LookupDomTree))
       return PreservedAnalyses::all();
     return PreservedAnalyses::none();
 }
 
 namespace {
 
 struct GlobalOptLegacyPass : public ModulePass {
   static char ID; // Pass identification, replacement for typeid
 
   GlobalOptLegacyPass() : ModulePass(ID) {
     initializeGlobalOptLegacyPassPass(*PassRegistry::getPassRegistry());
   }
 
   bool runOnModule(Module &M) override {
     if (skipModule(M))
       return false;
 
     auto &DL = M.getDataLayout();
     auto *TLI = &getAnalysis<TargetLibraryInfoWrapperPass>().getTLI();
     auto LookupDomTree = [this](Function &F) -> DominatorTree & {
       return this->getAnalysis<DominatorTreeWrapperPass>(F).getDomTree();
     };
-    return optimizeGlobalsInModule(M, DL, TLI, LookupDomTree);
+    auto GetTTI = [this](Function &F) -> TargetTransformInfo & {
+      return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
+    };
+
+    auto GetBFI = [this](Function &F) -> BlockFrequencyInfo & {
+      return this->getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI();
+    };
+
+    return optimizeGlobalsInModule(M, DL, TLI, GetTTI, GetBFI, LookupDomTree);
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addRequired<TargetLibraryInfoWrapperPass>();
+    AU.addRequired<TargetTransformInfoWrapperPass>();
+    AU.addRequired<BlockFrequencyInfoWrapperPass>();
     AU.addRequired<DominatorTreeWrapperPass>();
   }
 };
 
 } // end anonymous namespace
 
 char GlobalOptLegacyPass::ID = 0;
 
 INITIALIZE_PASS_BEGIN(GlobalOptLegacyPass, "globalopt",
                       "Global Variable Optimizer", false, false)
 INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(BlockFrequencyInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
 INITIALIZE_PASS_END(GlobalOptLegacyPass, "globalopt",
                     "Global Variable Optimizer", false, false)
 
 ModulePass *llvm::createGlobalOptimizerPass() {
   return new GlobalOptLegacyPass();
 }

test/CodeGen/PowerPC/coldcc.ll

+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu  < %s | FileCheck %s -check-prefix=COLDCC
+
+define signext i32 @caller(i32 signext %a, i32 signext %b, i32 signext %cold) {
+entry:
+  %0 = tail call i32 asm "add $0, $1, $2", "=r,r,r,~{r14},~{r15},~{r16},~{r17},~{r18},~{r19},~{r20},~{r21},~{r22},~{r23},~{r24},~{r25},~{r26},~{r27},~{r28},~{r29},~{r30},~{r31}"(i32 %a, i32 %b)
+  %mul = mul nsw i32 %0, %cold
+  %tobool = icmp eq i32 %cold, 0
+  br i1 %tobool, label %if.end, label %if.then
+
+if.then:                                          ; preds = %entry
+  %mul1 = mul nsw i32 %mul, %cold
+  %mul2 = mul nsw i32 %b, %a
+  %call = tail call coldcc signext i32 @callee(i32 signext %a, i32 signext %b)
+  %add = add i32 %mul2, %a
+  %add3 = add i32 %add, %mul
+  %add4 = add i32 %add3, %mul1
+  %add5 = add i32 %add4, %call
+  br label %if.end
+
+if.end:                                           ; preds = %entry, %if.then
+  %f.0 = phi i32 [ %add5, %if.then ], [ %0, %entry ]
+  ret i32 %f.0
+}
+
+define internal coldcc signext i32 @callee(i32 signext %a, i32 signext %b) local_unnamed_addr #0 {
+entry:
+; COLDCC: @callee
+; COLDCC: std 6, -8(1)
+; COLDCC: std 7, -16(1)
+; COLDCC: std 8, -24(1)
+; COLDCC: std 9, -32(1)
+; COLDCC: std 10, -40(1)
+; COLDCC: ld 9, -32(1)
+; COLDCC: ld 8, -24(1)
+; COLDCC: ld 7, -16(1)
+; COLDCC: ld 10, -40(1)
+; COLDCC: ld 6, -8(1)
+  %0 = tail call i32 asm "add $0, $1, $2", "=r,r,r,~{r6},~{r7},~{r8},~{r9},~{r10}"(i32 %a, i32 %b)
+  %mul = mul nsw i32 %a, 3
+  %1 = mul i32 %b, -5
+  %add = add i32 %1, %mul
+  %sub = add i32 %add, %0
+  ret i32 %sub
+}
+
+attributes #0 = { noinline }

test/CodeGen/PowerPC/coldcc2.ll

+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu < %s | FileCheck %s -check-prefix=COLDCC
+
+%struct.MyStruct = type { i32, i32, i32, i32 }
+
+@caller.s = internal unnamed_addr global %struct.MyStruct zeroinitializer, align 8
+
+define signext i32 @caller(i32 signext %a, i32 signext %b, i32 signext %cold) {
+entry:
+; COLDCC: bl callee
+; COLDCC: ld 4, 40(1)
+; COLDCC: ld 5, 32(1)
+  %call = tail call coldcc { i64, i64 } @callee(i32 signext %a, i32 signext %b)
+  %0 = extractvalue { i64, i64 } %call, 0
+  %1 = extractvalue { i64, i64 } %call, 1
+  store i64 %0, i64* bitcast (%struct.MyStruct* @caller.s to i64*), align 8
+  store i64 %1, i64* bitcast (i32* getelementptr inbounds (%struct.MyStruct, %struct.MyStruct* @caller.s, i64 0, i32 2) to i64*), align 8
+  %2 = lshr i64 %1, 32
+  %3 = trunc i64 %2 to i32
+  %sub = sub nsw i32 0, %3
+  ret i32 %sub
+}
+
+define internal coldcc { i64, i64 } @callee(i32 signext %a, i32 signext %b) {
+entry:
+; COLDCC: std {{[0-9]+}}, 0(3)
+; COLDCC: std {{[0-9]+}}, 8(3)
+  %0 = tail call i32 asm "add $0, $1, $2", "=r,r,r,~{r6},~{r7},~{r8},~{r9},~{r10}"(i32 %a, i32 %b)
+  %mul = mul nsw i32 %a, 3
+  %1 = mul i32 %b, -5
+  %add = add i32 %1, %mul
+  %sub = add i32 %add, %0
+  %mul5 = mul nsw i32 %b, %a
+  %add6 = add nsw i32 %sub, %mul5
+  %retval.sroa.0.0.insert.ext = zext i32 %0 to i64
+  %retval.sroa.3.8.insert.ext = zext i32 %sub to i64
+  %retval.sroa.3.12.insert.ext = zext i32 %add6 to i64
+  %retval.sroa.3.12.insert.shift = shl nuw i64 %retval.sroa.3.12.insert.ext, 32
+  %retval.sroa.3.12.insert.insert = or i64 %retval.sroa.3.12.insert.shift, %retval.sroa.3.8.insert.ext
+  %.fca.0.insert = insertvalue { i64, i64 } undef, i64 %retval.sroa.0.0.insert.ext, 0
+  %.fca.1.insert = insertvalue { i64, i64 } %.fca.0.insert, i64 %retval.sroa.3.12.insert.insert, 1
+  ret { i64, i64 } %.fca.1.insert
+}

test/Other/pass-pipelines.ll

 ; CHECK-O2: Loop Pass Manager
 ; CHECK-O2-NEXT: Loop Sink
 ; CHECK-O2: Simplify the CFG
 ; CHECK-O2-NOT: Manager
 ;
 ; FIXME: There really shouldn't be another pass manager, especially one that
 ; just builds the domtree. It doesn't even run the verifier.
 ; CHECK-O2: Pass Arguments:
-; CHECK-O2-NEXT: FunctionPass Manager
+; CHECK-O2: FunctionPass Manager
 ; CHECK-O2-NEXT: Dominator Tree Construction
 
 define void @foo() {
   ret void
 }

test/Transforms/GlobalOpt/coldcc_coldsites.ll

+; RUN: opt -globalopt -mtriple=powerpc64le-unknown-linux-gnu -ppc-enable-coldcc -S < %s | FileCheck %s -check-prefix=COLDCC
+; RUN: opt -globalopt -S < %s | FileCheck %s -check-prefix=CHECK
+
+define signext i32 @caller(i32 signext %a, i32 signext %b, i32 signext %lim, i32 signext %i) local_unnamed_addr #0 !prof !30 {
+entry:
+; COLDCC: call coldcc signext i32 @callee
+; CHECK: call fastcc signext i32 @callee
+  %add = add nsw i32 %b, %a
+  %sub = add nsw i32 %lim, -1
+  %cmp = icmp eq i32 %sub, %i
+  br i1 %cmp, label %if.then, label %if.end, !prof !31
+
+if.then:                                          ; preds = %entry
+  %call = tail call signext i32 @callee(i32 signext %a, i32 signext %b)
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %f.0 = phi i32 [ %call, %if.then ], [ %add, %entry ]
+  ret i32 %f.0
+}
+
+define internal signext i32 @callee(i32 signext %a, i32 signext %b) unnamed_addr #0 {
+entry:
+  %0 = tail call i32 asm "add $0, $1, $2", "=r,r,r,~{r6},~{r7},~{r8},~{r9}"(i32 %a, i32 %b) #1, !srcloc !32
+  %mul = mul nsw i32 %a, 3
+  %mul1 = shl i32 %0, 1
+  %add = add nsw i32 %mul1, %mul
+  ret i32 %add
+}
+
+define signext i32 @main() local_unnamed_addr #0 !prof !33 {
+entry:
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.body
+  %add.lcssa = phi i32 [ %add, %for.body ]
+  ret i32 %add.lcssa
+
+for.body:                                         ; preds = %for.body, %entry
+  %i.011 = phi i32 [ 0, %entry ], [ %inc, %for.body ]
+  %ret.010 = phi i32 [ 0, %entry ], [ %add, %for.body ]
+  %call = tail call signext i32 @caller(i32 signext 4, i32 signext 5, i32 signext 10000000, i32 signext %i.011)
+  %add = add nsw i32 %call, %ret.010
+  %inc = add nuw nsw i32 %i.011, 1
+  %exitcond = icmp eq i32 %inc, 10000000
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body, !prof !34
+}
+attributes #0 = { noinline }
+
+!0 = !{i32 1, !"ProfileSummary", !1}
+!1 = !{!2, !3, !4, !5, !6, !7, !8, !9}
+!2 = !{!"ProfileFormat", !"InstrProf"}
+!3 = !{!"TotalCount", i64 20000003}
+!4 = !{!"MaxCount", i64 10000000}
+!5 = !{!"MaxInternalCount", i64 10000000}
+!6 = !{!"MaxFunctionCount", i64 10000000}
+!7 = !{!"NumCounts", i64 5}
+!8 = !{!"NumFunctions", i64 3}
+!9 = !{!"DetailedSummary", !10}
+!10 = !{!11, !12, !13, !14, !15, !16, !16, !17, !17, !18, !19, !20, !21, !22, !23, !24, !25, !26}
+!11 = !{i32 10000, i64 10000000, i32 2}
+!12 = !{i32 100000, i64 10000000, i32 2}
+!13 = !{i32 200000, i64 10000000, i32 2}
+!14 = !{i32 300000, i64 10000000, i32 2}
+!15 = !{i32 400000, i64 10000000, i32 2}
+!16 = !{i32 500000, i64 10000000, i32 2}
+!17 = !{i32 600000, i64 10000000, i32 2}
+!18 = !{i32 700000, i64 10000000, i32 2}
+!19 = !{i32 800000, i64 10000000, i32 2}
+!20 = !{i32 900000, i64 10000000, i32 2}
+!21 = !{i32 950000, i64 10000000, i32 2}
+!22 = !{i32 990000, i64 10000000, i32 2}
+!23 = !{i32 999000, i64 10000000, i32 2}
+!24 = !{i32 999900, i64 10000000, i32 2}
+!25 = !{i32 999990, i64 10000000, i32 2}
+!26 = !{i32 999999, i64 10000000, i32 2}
+!30 = !{!"function_entry_count", i64 10000000}
+!31 = !{!"branch_weights", i32 2, i32 10000000}
+!32 = !{i32 59}
+!33 = !{!"function_entry_count", i64 1}
+!34 = !{!"branch_weights", i32 2, i32 10000001}

test/Transforms/GlobalOpt/coldcc_stress_test.ll

+; RUN: opt < %s -globalopt -S -enable-coldcc-stress-test -mtriple=powerpc64le-unknown-linux-gnu | FileCheck %s -check-prefix=COLDCC
+; RUN: opt < %s -globalopt -S | FileCheck %s -check-prefix=CHECK
+
+define internal i32 @callee_default(i32* %m) {
+; COLDCC-LABEL: define internal coldcc i32 @callee_default
+; CHECK-LABEL: define internal fastcc i32 @callee_default
+  %v = load i32, i32* %m
+  ret i32 %v
+}
+
+define internal fastcc i32 @callee_fastcc(i32* %m) {
+; COLDCC-LABEL: define internal fastcc i32 @callee_fastcc
+; CHECK-LABEL: define internal fastcc i32 @callee_fastcc
+  %v = load i32, i32* %m
+  ret i32 %v
+}
+
+define internal coldcc i32 @callee_coldcc(i32* %m) {
+; COLDCC-LABEL: define internal coldcc i32 @callee_coldcc
+; CHECK-LABEL: define internal coldcc i32 @callee_coldcc
+  %v = load i32, i32* %m
+  ret i32 %v
+}
+
+define i32 @callee(i32* %m) {
+  %v = load i32, i32* %m
+  ret i32 %v
+}
+
+define void @caller() {
+  %m = alloca i32
+  call i32 @callee_default(i32* %m)
+  call fastcc i32 @callee_fastcc(i32* %m)
+  call coldcc i32 @callee_coldcc(i32* %m)
+  call i32 @callee(i32* %m)
+  ret void
+}
+
+; COLDCC-LABEL: define void @caller()
+; COLDCC: call coldcc i32 @callee_default
+; COLDCC: call fastcc i32 @callee_fastcc
+; COLDCC: call coldcc i32 @callee_coldcc
+; COLDCC: call i32 @callee
+; CHECK-LABEL: define void @caller()
+; CHECK: call fastcc i32 @callee_default
+; CHECK: call fastcc i32 @callee_fastcc
+; CHECK: call coldcc i32 @callee_coldcc
+; CHECK: call i32 @callee