[Feedback requested] Implement cold spliting

[deadalnix]

Summary

This is an attempt at splitting cold code from regular code in a function. All landing pads are considered cold (in fact, they almost always are, especially if performance is a concern, and assuming this allow for simplification in the LSDA).

2 LSDA header and call sites are emitted, the cold one nested within the regular one. They both share the same action table and type table.

A symbol is emitted for the cold function, using the name of the function suffixed by $cold . Emitting a symbol is required when using .subsections_via_symbols .

Debug information aren't supported at this stage.

[davidxl]

There are quite a few refactoring changes that can be combined and split out. After that this patch can be minimized and becomes easier to review. Can you do the split first?

David

lib/CodeGen/AsmPrinter/DwarfCFIException.cpp

46

This can be split (with other refactoring changes) into a NFC patch.

138

Is it necessary to move this function above here? seems like an irrelevant change.

147

Another candidate of NFC refactoring.

lib/MC/MCObjectFileInfo.cpp

78

Should this section be created on demand when getColdTextSection() is called?

448

use .text.unlikely to be consistent with the name used in function reordering.

lib/MC/MCStreamer.cpp

258

This refactor change can go in its own patch. Also this change is not NFC -- the original code report fatal error regardless of whether NDEBUG is defined or not.

test/CodeGen/X86/coldsplit.ll

1

need mtriple (either linux or darwin) -- this does not work on COFF yet.

[mehdi_amini]

I guess the motivation is performance, right? Do you have benchmarks results to motivate this?

[MatzeB]

Very interesting, thanks for working on this!

I will do a detailed review soon. At a first glance it looks like this patch could be split up into a part affecting MC and one affecting the AsmPrinters? That would ease review and improve chances that some parts get accepted sooner.

In D17555#360358, @joker.eph wrote:

I guess the motivation is performance, right? Do you have benchmarks results to motivate this?

There is always the chance for less instruction caches misses if the cold code is out of the way. On top of this it should improve application startup time as there is a chance that the cold code will never be loaded from disk.

[escha]

This is a very clever idea. Getting code out of the way (WAY out of the way) can save not just icache, but iTLB too. I've seen large gains in JITs with this sort of strategy (e.g. putting extremely rarely taken code, like exception exits and things the guard intrinsic would be used for); just sticking the code in a separate, far-away allocation got significant improvements. If this can be even a fraction as effective on non-JITted code, it should be pretty nice.

[deadalnix]

@joker.eph , this is beneficial for performance for application that are icache and iTLB bound. This works is based on the various patches that were made to use LLVM as a backend for HHVM and was presented here : https://www.youtube.com/watch?v=VZ7A7t5LcR8 .

The optimization is disabled in the general case as it can have negative impact when you aren't icache/itlb bound (for instance bzip).

It may be worthwhile for instrumented build like ASAN and for one shot apps, but I haven't tested this, so don't quote me on this.

lib/CodeGen/AsmPrinter/DwarfCFIException.cpp

147

No, this need to be extracted as this is now needed twice: one for the regular fragment and once for the cold fragment.

lib/MC/MCObjectFileInfo.cpp

448

cold is the term used all over the place so far. It looks like GCC's crowd want to kill .text.unlikely on their side, so I'd advocate to keep it consistent and go for .cold , unless there is a good reason to stick with .unlikely ?

[davidxl]

lib/CodeGen/AsmPrinter/DwarfCFIException.cpp

147

that is what I am suggesting -- this part can be extracted into a helper function in another patch without changing functionality. This patch can then use it for cold fragment as well.

[davidxl]

Regarding .text.cold vs .text.unlikely, I am fine either way as long
as they are kept consistent.

Regarding performance impact, Teresa has done extensive tuning in the
past. Our experience is that with PGO, the compiler does a pretty good
job laying out hot BBs in long chains, so the function splitting's
impact on icache is not that significant. With huge page text, the
impact on TLB misses is also moderate. We do see some improvements
though.

thanks,

David

[reames]

FYI, this is something I'm interested in for the JIT use case. My code has lots of essentially never executed slow paths and getting them pulled far away from the normal code is interesting. Note that my case may be different that others in that the blocks I'm interested in pulling away are stone cold/never executed. This gives a clear profitability heuristic which is one of the complex parts of doing this for the general case.

[deadalnix]

test/CodeGen/X86/coldsplit.ll

1

Isn't the target trip in the module doing this already ?

[davidxl]

test/CodeGen/X86/coldsplit.ll

1

right -- but by extracting into command line, you can add RUN line for both ELF and MachO

[deadalnix]

Extracted some changes in http://reviews.llvm.org/D17579

test/CodeGen/X86/coldsplit.ll

1

Got you. Thanks.

[silvas]

I can see why this would help iTLB/paging, but I'm not grokking why it would help icache very much compared to per-function machine block placement ensuring that the cold stuff ends up at the end on a separate cacheline (does MBP already do that?). In fact (playing devil's advocate) the MBP approach could be more beneficial because it could allow branches to be relaxed to smaller encodings.

The scenarios I can see this being a substantial win for icache over MBP is when you e.g. have two functions with 1.5 cachelines of hot text (and say 1 cacheline of cold text). With MBP, each function would end up using ceiling(1.5) = 2 cachelines for the hot and one cacheline for the cold, but with the splitting the linker would see 2x 1.5 cacheline hot + 2x 1 cachline cold and so you could put the two 1.5's together and only use 3 cachelines for the hot part. How often does that occur (and does the linker actually manage to exploit this?).
Since the benefit is based on the "rounding", we save at most just under ("just under" is determined by the text alignment) one cacheline every time we can pack these densely. The benefit is at most #hotFunctions * (sizeof(Cacheline) - alignof(Function)) text size for the hot working set.

That being said, this kind of low-level function splitting is a really powerful tool and I fully support adding it, but I agree with Mehdi that I'd like to see some supporting benchmark results.

[Gerolf]

Can you also elaborate a bit more on the iCache benefits.Is this specific to an architecture or the JIT environment? Or part of a more elaborate implementation of function splitting than shared in the patch? Perhaps I'm missing something but I thought the splinter must be able to duplicate code to support code layout to reap Icache benefits. And a good block ordering algorithm is might already eat the benefits.

Also I'm curious about your design evaluation. I think the information you are looking at at the low level (at least currently) is also be available at the IR level. When the compiler could split there I can also see compile-time benefits eg. a function marked as cold would not have to be optimized. And there would be less code to optimize/analyze in the hot routines.

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

887

That deserves at least more comment and rational. Is this a good heuristic on all machines?

895

It looks weird that within the loop cold section is set, but never reset for a hot block. It seems that it would be cleaner to group all blocks into hot and cold.

977

I think when all block are grouped into hot and code the end_of_function directive could be issued at the same place for hot and cold section. It seems hard to mantain to have to think about hot and cold in various context.

[deadalnix]

@Gerolf The patch was originally made for LLVM's HHVM backend. The kind of code generated is very branchy, with a lot of cold branches, large and with a flat profile (ie most function are being used, but various path within these function are almost never taken). This optimization has proven to be very valuable for this kind of code. HHVM already uses large pages for JITed code, and even with this this is a valuable optimization. The effectiveness of the technique is heavily dependent on the type of code at hand, and, while it proves to be useful for HHVM, isn't in the general case.

I'd say if you have a large application, with a relatively flat profile, you may want to try this. If not then this is useless to you and may even hurt.

[deadalnix]

Rebased on top of some NFC patches, add triple in the llc call for tests rather than the module and test both linux and OSX.

[deadalnix]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

895

In practice it works, as this ends up being more or less what you get out of the MachineBlockPlacement pass. Ideally, that's be indeed preferable that MachineBlockPlacement flag a BB after which all BB are cold or something.

Having one split is preferable, as well as having all exception unwinding related code is the same fragment. You don't want to jump back and forth between cold and hot code, and generating LSDA would become way too hairy =.

977

I'm not sure what you mean here. Cold code is physically separated from hot, so you need symbols to express ranges in both.

[lhames]

Neat - this could improve compile-times for lazily compiled functions in the ORC JIT. Thanks for working on this. :)

[• rafael]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

901

At least on ELF you should not need both symbols. I would suggest just passing the function name to createTempSymbol.

Even on MachO you should be able to use a single linkerPrivate symbol, no?

[davidxl]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

875

Should the creation of the ColdTextSection be guarded with the option such that if it is not enabled, null pointer will be returned and checked here?

888

This is not correct -- it will trigger debug assert when MBB's frequency is larger than the Entry's -- skip that case first. Also you are using BP as a ratio here not really as branch probability.

BBFreq = ...;

IsCold = (BBFreq < EntryFreq && BranchProbablity::getBranchProbablity(BBFreq, EntryFreq) <= BranchProbability(1, 1<<...);

895

This does not look like a good assumption to make about MBP. MBP does outline cold blocks from loops and layout them last, but not all such blocks are suitable to be split out.

[deadalnix]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

895

I'm working toward MBP being able to flag a block. Note that MBP already outline blocks in loop if they run < 20% of the time.

include/llvm/CodeGen/AsmPrinter.h

public:
   typedef std::pair<const GlobalVariable *, unsigned> GOTEquivUsePair;
   MapVector<const MCSymbol *, GOTEquivUsePair> GlobalGOTEquivs;
 
 private:
   MCSymbol *CurrentFnBegin;
   MCSymbol *CurrentFnEnd;
   MCSymbol *CurExceptionSym;
 
+  MCSymbol *CurrentFnColdBegin;
+  MCSymbol *CurrentFnColdEnd;
+  MCSymbol *CurColdExceptionSym;
+
+  const MachineBasicBlock *ColdFragmentStart;
+
   // The garbage collection metadata printer table.
   void *GCMetadataPrinters; // Really a DenseMap.
 
   /// Emit comments in assembly output if this is true.
   ///
   bool VerboseAsm;
   static char ID;
 
public:
 
   /// Return a unique ID for the current function.
   ///
   unsigned getFunctionNumber() const;
 
   MCSymbol *getFunctionBegin() const { return CurrentFnBegin; }
   MCSymbol *getFunctionEnd() const { return CurrentFnEnd; }
   MCSymbol *getCurExceptionSym();
+  MCSymbol *getFunctionColdBegin() const { return CurrentFnColdBegin; }
+  MCSymbol *getFunctionColdEnd() const { return CurrentFnColdEnd; }
+  MCSymbol *getCurColdExceptionSym();
+
+  const MachineBasicBlock *getColdFragmentStart() const {
+    return ColdFragmentStart;
+  }
 
   /// Return information about object file lowering.
   const TargetLoweringObjectFile &getObjFileLowering() const;
 
   /// Return information about data layout.
   const DataLayout &getDataLayout() const;
 
   /// Return the pointer size from the TargetMachine

include/llvm/MC/MCObjectFileInfo.h

protected:
   unsigned TTypeEncoding;
 
   /// Compact unwind encoding indicating that we should emit only an EH frame.
   unsigned CompactUnwindDwarfEHFrameOnly;
 
   /// Section directive for standard text.
   MCSection *TextSection;
 
+  /// Section directive for cold text.
+  MCSection *ColdTextSection;
+
   /// Section directive for standard data.
   MCSection *DataSection;
 
   /// Section that is default initialized to zero.
   MCSection *BSSSection;
 
   /// Section that is readonly and can contain arbitrary initialized data.
   /// Targets are not required to have a readonly section. If they don't,
public:
   unsigned getFDEEncoding() const { return FDECFIEncoding; }
   unsigned getTTypeEncoding() const { return TTypeEncoding; }
 
   unsigned getCompactUnwindDwarfEHFrameOnly() const {
     return CompactUnwindDwarfEHFrameOnly;
   }
 
   MCSection *getTextSection() const { return TextSection; }
+  MCSection *getColdTextSection() const { return ColdTextSection; }
   MCSection *getDataSection() const { return DataSection; }
   MCSection *getBSSSection() const { return BSSSection; }
   MCSection *getReadOnlySection() const { return ReadOnlySection; }
   MCSection *getLSDASection() const { return LSDASection; }
   MCSection *getCompactUnwindSection() const { return CompactUnwindSection; }
   MCSection *getDwarfAbbrevSection() const { return DwarfAbbrevSection; }
   MCSection *getDwarfInfoSection() const { return DwarfInfoSection; }
   MCSection *getDwarfLineSection() const { return DwarfLineSection; }

include/llvm/MC/MCStreamer.h

MCTargetStreamer *getTargetStreamer() {
     return TargetStreamer.get();
   }
 
   unsigned getNumFrameInfos() { return DwarfFrameInfos.size(); }
   ArrayRef<MCDwarfFrameInfo> getDwarfFrameInfos() const {
     return DwarfFrameInfos;
   }
 
+  bool hasUnfinishedDwarfFrameInfo();
+  MCDwarfFrameInfo popDwarfFrameInfo();
+  void pushDwarfFrameInfo(MCDwarfFrameInfo DFI);
+
   unsigned getNumWinFrameInfos() { return WinFrameInfos.size(); }
   ArrayRef<WinEH::FrameInfo *> getWinFrameInfos() const {
     return WinFrameInfos;
   }
 
   void generateCompactUnwindEncodings(MCAsmBackend *MAB);
 
   /// \name Assembly File Formatting.

include/llvm/MC/MCTargetOptions.h

public:
   bool MCFatalWarnings : 1;
   bool MCNoWarn : 1;
   bool MCSaveTempLabels : 1;
   bool MCUseDwarfDirectory : 1;
   bool MCIncrementalLinkerCompatible : 1;
   bool ShowMCEncoding : 1;
   bool ShowMCInst : 1;
   bool AsmVerbose : 1;
+  bool SplitColdCode : 1;
   int DwarfVersion;
   /// getABIName - If this returns a non-empty string this represents the
   /// textual name of the ABI that we want the backend to use, e.g. o32, or
   /// aapcs-linux.
   StringRef getABIName() const;
   std::string ABIName;
   MCTargetOptions();
 };
 
 inline bool operator==(const MCTargetOptions &LHS, const MCTargetOptions &RHS) {
 #define ARE_EQUAL(X) LHS.X == RHS.X
   return (ARE_EQUAL(SanitizeAddress) &&
           ARE_EQUAL(MCRelaxAll) &&
           ARE_EQUAL(MCNoExecStack) &&
           ARE_EQUAL(MCFatalWarnings) &&
           ARE_EQUAL(MCNoWarn) &&
           ARE_EQUAL(MCSaveTempLabels) &&
           ARE_EQUAL(MCUseDwarfDirectory) &&
           ARE_EQUAL(MCIncrementalLinkerCompatible) &&
           ARE_EQUAL(ShowMCEncoding) &&
           ARE_EQUAL(ShowMCInst) &&
           ARE_EQUAL(AsmVerbose) &&
+          ARE_EQUAL(SplitColdCode) &&
           ARE_EQUAL(DwarfVersion) &&
           ARE_EQUAL(ABIName));
 #undef ARE_EQUAL
 }
 
 inline bool operator!=(const MCTargetOptions &LHS, const MCTargetOptions &RHS) {
   return !(LHS == RHS);
 }
 
 } // end namespace llvm
 
 #endif

include/llvm/MC/MCTargetOptionsCommandFlags.h

 
 cl::opt<int> DwarfVersion("dwarf-version", cl::desc("Dwarf version"),
                           cl::init(0));
 
 cl::opt<bool> ShowMCInst("asm-show-inst",
                          cl::desc("Emit internal instruction representation to "
                                   "assembly file"));
 
+cl::opt<bool> SplitColdCode("split-cold-code",
+                            cl::desc("Emit cold code in a different section"));
+
 cl::opt<bool> FatalWarnings("fatal-warnings",
                             cl::desc("Treat warnings as errors"));
 
 cl::opt<bool> NoWarn("no-warn", cl::desc("Suppress all warnings"));
 cl::alias NoWarnW("W", cl::desc("Alias for --no-warn"), cl::aliasopt(NoWarn));
 
 cl::opt<std::string>
 ABIName("target-abi", cl::Hidden,
         cl::desc("The name of the ABI to be targeted from the backend."),
         cl::init(""));
 
 static inline MCTargetOptions InitMCTargetOptionsFromFlags() {
   MCTargetOptions Options;
   Options.SanitizeAddress =
       (AsmInstrumentation == MCTargetOptions::AsmInstrumentationAddress);
   Options.MCRelaxAll = RelaxAll;
   Options.MCIncrementalLinkerCompatible = IncrementalLinkerCompatible;
   Options.DwarfVersion = DwarfVersion;
   Options.ShowMCInst = ShowMCInst;
   Options.ABIName = ABIName;
+  Options.SplitColdCode = SplitColdCode;
   Options.MCFatalWarnings = FatalWarnings;
   Options.MCNoWarn = NoWarn;
   return Options;
 }
 
 #endif

include/llvm/MC/SectionKind.h

 class SectionKind {
   enum Kind {
     /// Metadata - Debug info sections or other metadata.
     Metadata,
 
     /// Text - Text section, used for functions and other executable code.
     Text,
 
+    /// Text - Text section, used for functions and other rarely executed code.
+    ColdText,
+
     /// ReadOnly - Data that is never written to at program runtime by the
     /// program or the dynamic linker.  Things in the top-level readonly
     /// SectionKind are not mergeable.
     ReadOnly,
 
         /// MergableCString - Any null-terminated string which allows merging.
         /// These values are known to end in a nul value of the specified size,
         /// not otherwise contain a nul value, and be mergable.  This allows the
/// to during program runtime.
            /// mark the pages these globals end up on as read-only after it is
            /// done with its relocation phase.
            ReadOnlyWithRel
   } K : 8;
 public:
 
   bool isMetadata() const { return K == Metadata; }
   bool isText() const { return K == Text; }
+  bool isColdText() const { return K == ColdText; }
 
   bool isReadOnly() const {
     return K == ReadOnly || isMergeableCString() ||
            isMergeableConst();
   }
 
   bool isMergeableCString() const {
     return K == Mergeable1ByteCString || K == Mergeable2ByteCString ||
static SectionKind get(Kind K) {
     SectionKind Res;
     Res.K = K;
     return Res;
   }
 public:
 
   static SectionKind getMetadata() { return get(Metadata); }
   static SectionKind getText() { return get(Text); }
+  static SectionKind getColdText() { return get(ColdText); }
   static SectionKind getReadOnly() { return get(ReadOnly); }
   static SectionKind getMergeable1ByteCString() {
     return get(Mergeable1ByteCString);
   }
   static SectionKind getMergeable2ByteCString() {
     return get(Mergeable2ByteCString);
   }
   static SectionKind getMergeable4ByteCString() {

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

 #include "DwarfException.h"
 #include "WinException.h"
 #include "CodeViewDebug.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/ConstantFolding.h"
 #include "llvm/CodeGen/Analysis.h"
 #include "llvm/CodeGen/GCMetadataPrinter.h"
+#include "llvm/CodeGen/MachineBlockFrequencyInfo.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstrBundle.h"
 #include "llvm/CodeGen/MachineJumpTableInfo.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/CodeGen/MachineModuleInfoImpls.h"
 #include "llvm/IR/DataLayout.h"
: MachineFunctionPass(ID), TM(tm), MAI(tm.getMCAsmInfo()),
       LastMI(nullptr), LastFn(0), Counter(~0U) {
   DD = nullptr;
   MMI = nullptr;
   LI = nullptr;
   MF = nullptr;
   CurExceptionSym = CurrentFnSym = CurrentFnSymForSize = nullptr;
   CurrentFnBegin = nullptr;
   CurrentFnEnd = nullptr;
+  CurrentFnColdBegin = nullptr;
+  CurrentFnColdEnd = nullptr;
+  ColdFragmentStart = nullptr;
   GCMetadataPrinters = nullptr;
   VerboseAsm = OutStreamer->isVerboseAsm();
 }
 
 AsmPrinter::~AsmPrinter() {
   assert(!DD && Handlers.empty() && "Debug/EH info didn't get finalized");
 
   if (GCMetadataPrinters) {

 
 void AsmPrinter::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.setPreservesAll();
   MachineFunctionPass::getAnalysisUsage(AU);
   AU.addRequired<MachineModuleInfo>();
   AU.addRequired<GCModuleInfo>();
   if (isVerbose())
     AU.addRequired<MachineLoopInfo>();
+  if (TM.Options.MCOptions.SplitColdCode)
+    AU.addRequired<MachineBlockFrequencyInfo>();
 }
 
 bool AsmPrinter::doInitialization(Module &M) {
   MMI = getAnalysisIfAvailable<MachineModuleInfo>();
 
   // Initialize TargetLoweringObjectFile.
   const_cast<TargetLoweringObjectFile&>(getObjFileLowering())
     .Initialize(OutContext, TM);
void AsmPrinter::emitFrameAlloc(const MachineInstr &MI) {
   MCSymbol *FrameAllocSym = MI.getOperand(0).getMCSymbol();
   int FrameOffset = MI.getOperand(1).getImm();
 
   // Emit a symbol assignment.
   OutStreamer->EmitAssignment(FrameAllocSym,
                              MCConstantExpr::create(FrameOffset, OutContext));
 }
 
+static bool isReachedByFallthrough(const MachineBasicBlock &MBB) {
+  // The first BB cannot be accessed via fallthrough.
+  if (&*MBB.getParent()->begin() == &MBB)
+    return false;
+  MachineFunction::iterator I(const_cast<MachineBasicBlock*>(&MBB));
+  return (--I)->canFallThrough();
+}
+
+static MCSymbol *getColdExceptionSym(AsmPrinter *Asm) {
+  return Asm->getCurColdExceptionSym();
+}
+
 /// EmitFunctionBody - This method emits the body and trailer for a
 /// function.
 void AsmPrinter::EmitFunctionBody() {
   EmitFunctionHeader();
 
   // Emit target-specific gunk before the function body.
   EmitFunctionBodyStart();
 
   bool ShouldPrintDebugScopes = MMI->hasDebugInfo();
 
+  auto *HotSection = const_cast<MCSection*>(getCurrentSection());
+  auto *ColdSection = getObjFileLowering().getColdTextSection();
+  auto *CurSection = HotSection;
+
+  MachineBlockFrequencyInfo *MBFI;
+  uint64_t EntryFreq;
+
+  bool SplitColdCode = ColdSection && TM.Options.MCOptions.SplitColdCode;
+  if (SplitColdCode) {
+    MBFI = &getAnalysis<MachineBlockFrequencyInfo>();
+    EntryFreq = MBFI->getEntryFreq();
+  }
+
   // Print out code for the function.
   bool HasAnyRealCode = false;
   for (auto &MBB : *MF) {
+    if (SplitColdCode && CurSection != ColdSection &&
+        !isReachedByFallthrough(MBB)) {
+      bool IsCold = MBB.isEHPad();
+      if (!IsCold) {
+        auto BP = BranchProbability::getBranchProbability(
+                        MBFI->getBlockFreq(&MBB).getFrequency(), EntryFreq);
+        uint64_t ColdThresold = 1 << 14;
+        IsCold = BP.getNumerator() < ColdThresold;
+      }
+
+      if (IsCold) {
+        CurSection = ColdSection;
+        ColdFragmentStart = &MBB;
+        OutStreamer->SwitchSection(ColdSection);
+        MCSymbol *ColdFnSym =
+          OutContext.getOrCreateSymbol(CurrentFnSym->getName() + "$cold");
+        OutStreamer->EmitLabel(ColdFnSym);
+        CurrentFnColdBegin = createTempSymbol("func_cold_begin");
+        OutStreamer->EmitLabel(CurrentFnColdBegin);
+
+        for (const HandlerInfo &HI : Handlers) {
+          HI.Handler->beginFragment(&MBB, getColdExceptionSym);
+        }
+      }
+    }
+
     // Print a label for the basic block.
     EmitBasicBlockStart(MBB);
     for (auto &MI : MBB) {
 
       // Print the assembly for the instruction.
       if (!MI.isPosition() && !MI.isImplicitDef() && !MI.isKill() &&
           !MI.isDebugValue()) {
         HasAnyRealCode = true;
for (auto &MI : MBB) {
           HI.Handler->endInstruction();
         }
       }
     }
 
     EmitBasicBlockEnd(MBB);
   }
 
+  if (CurSection == ColdSection) {
+    CurSection = HotSection;
+    CurrentFnColdEnd = createTempSymbol("func_cold_end");
+    OutStreamer->EmitLabel(CurrentFnColdEnd);
+    for (const HandlerInfo &HI : Handlers) {
+      HI.Handler->endFragment();
+    }
+
+    OutStreamer->SwitchSection(HotSection);
+  }
+
   // If the function is empty and the object file uses .subsections_via_symbols,
   // then we need to emit *something* to the function body to prevent the
   // labels from collapsing together.  Just emit a noop.
   if ((MAI->hasSubsectionsViaSymbols() && !HasAnyRealCode)) {
     MCInst Noop;
     MF->getSubtarget().getInstrInfo()->getNoopForMachoTarget(Noop);
     OutStreamer->AddComment("avoids zero-length function");
 

 }
 
 MCSymbol *AsmPrinter::getCurExceptionSym() {
   if (!CurExceptionSym)
     CurExceptionSym = createTempSymbol("exception");
   return CurExceptionSym;
 }
 
+MCSymbol *AsmPrinter::getCurColdExceptionSym() {
+  if (!CurColdExceptionSym)
+    CurColdExceptionSym = createTempSymbol("cold_exception");
+  return CurColdExceptionSym;
+}
+
 void AsmPrinter::SetupMachineFunction(MachineFunction &MF) {
   this->MF = &MF;
   // Get the function symbol.
   CurrentFnSym = getSymbol(MF.getFunction());
   CurrentFnSymForSize = CurrentFnSym;
   CurrentFnBegin = nullptr;
   CurExceptionSym = nullptr;
   bool NeedsLocalForSize = MAI->needsLocalForSize();

lib/CodeGen/AsmPrinter/AsmPrinterHandler.h

 
 #ifndef LLVM_LIB_CODEGEN_ASMPRINTER_ASMPRINTERHANDLER_H
 #define LLVM_LIB_CODEGEN_ASMPRINTER_ASMPRINTERHANDLER_H
 
 #include "llvm/Support/DataTypes.h"
 
 namespace llvm {
 
+class AsmPrinter;
 class MachineBasicBlock;
 class MachineFunction;
 class MachineInstr;
 class MCSymbol;
 
+typedef MCSymbol *ExceptionSymbolProvider(AsmPrinter *Asm);
+
 /// \brief Collects and handles AsmPrinter objects required to build debug
 /// or EH information.
 class AsmPrinterHandler {
 public:
   virtual ~AsmPrinterHandler();
 
   /// \brief For symbols that have a size designated (e.g. common symbols),
   /// this tracks that size.
public:
   // before endFunction and cannot switch sections.
   virtual void markFunctionEnd();
 
   /// \brief Gather post-function debug information.
   /// Please note that some AsmPrinter implementations may not call
   /// beginFunction at all.
   virtual void endFunction(const MachineFunction *MF) = 0;
 
+  virtual void beginFragment(const MachineBasicBlock *MBB,
+                             ExceptionSymbolProvider ESP) {}
+  virtual void endFragment() {}
+
   /// \brief Emit target-specific EH funclet machinery.
   virtual void beginFunclet(const MachineBasicBlock &MBB,
                             MCSymbol *Sym = nullptr) {}
   virtual void endFunclet() {}
 
   /// \brief Process beginning of an instruction.
   virtual void beginInstruction(const MachineInstr *MI) = 0;
 
   /// \brief Process end of an instruction.
   virtual void endInstruction() = 0;
 };
 } // End of namespace llvm
 
 #endif

lib/CodeGen/AsmPrinter/DwarfCFIException.cpp

 #include "llvm/Target/TargetOptions.h"
 #include "llvm/Target/TargetRegisterInfo.h"
 using namespace llvm;
 
 DwarfCFIExceptionBase::DwarfCFIExceptionBase(AsmPrinter *A)
     : EHStreamer(A), shouldEmitCFI(false) {}
 
 void DwarfCFIExceptionBase::markFunctionEnd() {
-  if (shouldEmitCFI)
-    Asm->OutStreamer->EmitCFIEndProc();
+  endFragment();
 
   if (MMI->getLandingPads().empty())
     return;
 
   // Map all labels and get rid of any dead landing pads.
   MMI->TidyLandingPads();
 }
 
void DwarfCFIException::endModule() {
   for (const Function *Personality : MMI->getPersonalities()) {
     if (!Personality)
       continue;
     MCSymbol *Sym = Asm->getSymbol(Personality);
     TLOF.emitPersonalityValue(*Asm->OutStreamer, Asm->getDataLayout(), Sym);
   }
 }
 
+static MCSymbol *getRegularExceptionSym(AsmPrinter *Asm) {
+  return Asm->getCurExceptionSym();
+}
+
 void DwarfCFIException::beginFunction(const MachineFunction *MF) {
   shouldEmitMoves = shouldEmitPersonality = shouldEmitLSDA = false;
   const Function *F = MF->getFunction();
 
   // If any landing pads survive, we need an EH table.
   bool hasLandingPads = !MMI->getLandingPads().empty();
 
   // See if we need frame move info.
   AsmPrinter::CFIMoveType MoveType = Asm->needsCFIMoves();
   if (MoveType == AsmPrinter::CFI_M_EH ||
       (MoveType == AsmPrinter::CFI_M_Debug &&
        moveTypeModule == AsmPrinter::CFI_M_None))
     moveTypeModule = MoveType;
 
   shouldEmitMoves = MoveType != AsmPrinter::CFI_M_None;
 
   const TargetLoweringObjectFile &TLOF = Asm->getObjFileLowering();
   unsigned PerEncoding = TLOF.getPersonalityEncoding();
   const Function *Per = nullptr;
   if (F->hasPersonalityFn())
     Per = dyn_cast<Function>(F->getPersonalityFn()->stripPointerCasts());
 
   // Emit a personality function even when there are no landing pads
-  bool forceEmitPersonality =
+  forceEmitPersonality =
       // ...if a personality function is explicitly specified
       F->hasPersonalityFn() &&
       // ... and it's not known to be a noop in the absence of invokes
       !isNoOpWithoutInvoke(classifyEHPersonality(Per)) &&
       // ... and we're not explicitly asked not to emit it
       F->needsUnwindTableEntry();
 
   shouldEmitPersonality =
       (forceEmitPersonality ||
        (hasLandingPads && PerEncoding != dwarf::DW_EH_PE_omit)) &&
       Per;
 
   unsigned LSDAEncoding = TLOF.getLSDAEncoding();
   shouldEmitLSDA = shouldEmitPersonality &&
     LSDAEncoding != dwarf::DW_EH_PE_omit;
 
   shouldEmitCFI = shouldEmitPersonality || shouldEmitMoves;
+  beginFragment(&*MF->begin(), getRegularExceptionSym);
+}
+
+/// endFunction - Gather and emit post-function exception information.
+///
+void DwarfCFIException::endFunction(const MachineFunction *) {
+  assert(InFlightCFIs.empty() && "Remaining inflight CFI");
+
+  if (!shouldEmitPersonality)
+    return;
+
+  emitExceptionTable();
+}
+
+void DwarfCFIException::beginFragment(const MachineBasicBlock *MBB,
+                                      ExceptionSymbolProvider ESP) {
   if (!shouldEmitCFI)
     return;
 
+  bool IsSubFragment = Asm->OutStreamer->hasUnfinishedDwarfFrameInfo();
+  if (IsSubFragment)
+    InFlightCFIs.push_back(Asm->OutStreamer->popDwarfFrameInfo());
+
   Asm->OutStreamer->EmitCFIStartProc(/*IsSimple=*/false);
 
   // Indicate personality routine, if any.
-  if (!shouldEmitPersonality)
-    return;
+  if (shouldEmitPersonality) {
+    auto *F = MBB->getParent()->getFunction();
+    auto *P = dyn_cast<Function>(F->getPersonalityFn()->stripPointerCasts());
+    assert(P && "Expected personality function");
 
     // If we are forced to emit this personality, make sure to record
     // it because it might not appear in any landingpad
     if (forceEmitPersonality)
-    MMI->addPersonality(Per);
+      MMI->addPersonality(P);
 
+    const TargetLoweringObjectFile &TLOF = Asm->getObjFileLowering();
+    unsigned PerEncoding = TLOF.getPersonalityEncoding();
     const MCSymbol *Sym =
-      TLOF.getCFIPersonalitySymbol(Per, *Asm->Mang, Asm->TM, MMI);
+        TLOF.getCFIPersonalitySymbol(P, *Asm->Mang, Asm->TM, MMI);
     Asm->OutStreamer->EmitCFIPersonality(Sym, PerEncoding);
 
     // Provide LSDA information.
-  if (!shouldEmitLSDA)
-    return;
-
-  Asm->OutStreamer->EmitCFILsda(Asm->getCurExceptionSym(), LSDAEncoding);
+    if (shouldEmitLSDA)
+      Asm->OutStreamer->EmitCFILsda(ESP(Asm), TLOF.getLSDAEncoding());
   }
 
-/// endFunction - Gather and emit post-function exception information.
-///
-void DwarfCFIException::endFunction(const MachineFunction *) {
-  if (!shouldEmitPersonality)
-    return;
+  // There must be a better way to do this, but it will do for now.
+  if (IsSubFragment)
+    for (const MCCFIInstruction &I : InFlightCFIs.back().Instructions)
+      Asm->emitCFIInstruction(I);
+}
 
-  emitExceptionTable();
+void DwarfCFIException::endFragment() {
+  if (shouldEmitCFI)
+    Asm->OutStreamer->EmitCFIEndProc();
+  if (InFlightCFIs.size())
+    Asm->OutStreamer->pushDwarfFrameInfo(InFlightCFIs.pop_back_val());
 }

lib/CodeGen/AsmPrinter/DwarfException.h

 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIB_CODEGEN_ASMPRINTER_DWARFEXCEPTION_H
 #define LLVM_LIB_CODEGEN_ASMPRINTER_DWARFEXCEPTION_H
 
 #include "EHStreamer.h"
 #include "llvm/CodeGen/AsmPrinter.h"
+#include "llvm/MC/MCDwarf.h"
 
 namespace llvm {
 class MachineFunction;
 class ARMTargetStreamer;
 
 class LLVM_LIBRARY_VISIBILITY DwarfCFIExceptionBase : public EHStreamer {
 protected:
   DwarfCFIExceptionBase(AsmPrinter *A);
 
   /// Per-function flag to indicate if frame CFI info should be emitted.
   bool shouldEmitCFI;
 
   void markFunctionEnd() override;
 };
 
 class LLVM_LIBRARY_VISIBILITY DwarfCFIException : public DwarfCFIExceptionBase {
   /// Per-function flag to indicate if .cfi_personality should be emitted.
   bool shouldEmitPersonality;
 
+  /// Per-function flag to indicate if .cfi_personality must be emitted.
+  bool forceEmitPersonality;
+
   /// Per-function flag to indicate if .cfi_lsda should be emitted.
   bool shouldEmitLSDA;
 
   /// Per-function flag to indicate if frame moves info should be emitted.
   bool shouldEmitMoves;
 
   AsmPrinter::CFIMoveType moveTypeModule;
 
+  SmallVector<MCDwarfFrameInfo, 2> InFlightCFIs;
+
 public:
   //===--------------------------------------------------------------------===//
   // Main entry points.
   //
   DwarfCFIException(AsmPrinter *A);
   ~DwarfCFIException() override;
 
   /// Emit all exception information that should come after the content.
   void endModule() override;
 
   /// Gather pre-function exception information.  Assumes being emitted
   /// immediately after the function entry point.
   void beginFunction(const MachineFunction *MF) override;
 
   /// Gather and emit post-function exception information.
   void endFunction(const MachineFunction *) override;
+
+  void beginFragment(const MachineBasicBlock *MBB,
+                     ExceptionSymbolProvider ESP) override;
+  void endFragment() override;
 };
 
 class LLVM_LIBRARY_VISIBILITY ARMException : public DwarfCFIExceptionBase {
   void emitTypeInfos(unsigned TTypeEncoding) override;
   ARMTargetStreamer &getTargetStreamer();
 
 public:
   //===--------------------------------------------------------------------===//

lib/CodeGen/AsmPrinter/EHStreamer.h

 #include "AsmPrinterHandler.h"
 #include "llvm/ADT/DenseMap.h"
 
 namespace llvm {
 struct LandingPadInfo;
 class MachineModuleInfo;
 class MachineInstr;
 class MachineFunction;
-class AsmPrinter;
 class MCSymbol;
 class MCSymbolRefExpr;
 
 template <typename T>
 class SmallVectorImpl;
 
 /// Emits exception handling directives.
 class LLVM_LIBRARY_VISIBILITY EHStreamer : public AsmPrinterHandler {
void computePadMap(const SmallVectorImpl<const LandingPadInfo *> &LandingPads,
                      RangeMapType &PadMap);
 
   /// Compute the call-site table.  The entry for an invoke has a try-range
   /// containing the call, a non-zero landing pad and an appropriate action.
   /// The entry for an ordinary call has a try-range containing the call and
   /// zero for the landing pad and the action.  Calls marked 'nounwind' have
   /// no entry and must not be contained in the try-range of any entry - they
   /// form gaps in the table.  Entries must be ordered by try-range address.
-  void computeCallSiteTable(SmallVectorImpl<CallSiteEntry> &CallSites,
+  unsigned computeCallSiteTable(SmallVectorImpl<CallSiteEntry> &CallSites,
                                 const SmallVectorImpl<const LandingPadInfo *> &LPs,
                                 const SmallVectorImpl<unsigned> &FirstActions);
 
   /// Emit landing pads and actions.
   ///
   /// The general organization of the table is complex, but the basic concepts
   /// are easy.  First there is a header which describes the location and
   /// organization of the three components that follow.
   ///  1. The landing pad site information describes the range of code covered
   ///     by the try.  In our case it's an accumulation of the ranges covered
   ///     by the invokes in the try.  There is also a reference to the landing
   ///     pad that handles the exception once processed.  Finally an index into
   ///     the actions table.
   ///  2. The action table, in our case, is composed of pairs of type ids
   ///     and next action offset.  Starting with the action index from the
   ///     landing pad site, each type Id is checked for a match to the current
   ///     exception.  If it matches then the exception and type id are passed
   ///     on to the landing pad.  Otherwise the next action is looked up.  This
   ///     chain is terminated with a next action of zero.  If no type id is
   ///     found the frame is unwound and handling continues.
   ///  3. Type id table contains references to all the C++ typeinfo for all
   ///     catches in the function.  This tables is reversed indexed base 1.
   void emitExceptionTable();
 
+  void emitLSDAHeader(MCSymbol *LPStart, unsigned SubCallSiteTableLength,
+                      unsigned CallSiteTableLength, unsigned TTypeEncoding,
+                      unsigned SizeActionsAndTypes);
+
   virtual void emitTypeInfos(unsigned TTypeEncoding);
 
   // Helpers for for identifying what kind of clause an EH typeid or selector
   // corresponds to. Negative selectors are for filter clauses, the zero
   // selector is for cleanups, and positive selectors are for catch clauses.
   static bool isFilterEHSelector(int Selector) { return Selector < 0; }
   static bool isCleanupEHSelector(int Selector) { return Selector == 0; }
   static bool isCatchEHSelector(int Selector) { return Selector > 0; }

lib/CodeGen/AsmPrinter/EHStreamer.cpp

 }
 
 /// Compute the call-site table.  The entry for an invoke has a try-range
 /// containing the call, a non-zero landing pad, and an appropriate action.  The
 /// entry for an ordinary call has a try-range containing the call and zero for
 /// the landing pad and the action.  Calls marked 'nounwind' have no entry and
 /// must not be contained in the try-range of any entry - they form gaps in the
 /// table.  Entries must be ordered by try-range address.
-void EHStreamer::
+unsigned EHStreamer::
 computeCallSiteTable(SmallVectorImpl<CallSiteEntry> &CallSites,
                      const SmallVectorImpl<const LandingPadInfo *> &LandingPads,
                      const SmallVectorImpl<unsigned> &FirstActions) {
   RangeMapType PadMap;
   computePadMap(LandingPads, PadMap);
 
   // The end label of the previous invoke or nounwind try-range.
   MCSymbol *LastLabel = nullptr;
 
   // Whether there is a potentially throwing instruction (currently this means
   // an ordinary call) between the end of the previous try-range and now.
   bool SawPotentiallyThrowing = false;
 
   // Whether the last CallSite entry was for an invoke.
   bool PreviousIsInvoke = false;
 
   bool IsSJLJ = Asm->MAI->getExceptionHandlingType() == ExceptionHandling::SjLj;
+  bool SplitColdCode = Asm->getFunctionColdBegin() != nullptr;
+  bool IsInCold = false;
+
+  unsigned RegularCallCount = 0;
 
   // Visit all instructions in order of address.
   for (const auto &MBB : *Asm->MF) {
+    if (!IsInCold && SplitColdCode && LastLabel && Asm->getColdFragmentStart() == &MBB) {
+      auto FnEnd = Asm->getFunctionEnd();
+      CallSiteEntry Site = { LastLabel, FnEnd, nullptr, 0 };
+      CallSites.push_back(Site);
+      RegularCallCount++;
+      SawPotentiallyThrowing = false;
+      PreviousIsInvoke = false;
+      IsInCold = true;
+      LastLabel = Asm->getFunctionColdBegin();
+    }
+
     for (const auto &MI : MBB) {
       if (!MI.isEHLabel()) {
         if (MI.isCall())
           SawPotentiallyThrowing |= !callToNoUnwindFunction(&MI);
         continue;
       }
 
       // End of the previous try-range?
for (const auto &MI : MBB) {
       // For Dwarf exception handling (SjLj handling doesn't use this). If some
       // instruction between the previous try-range and this one may throw,
       // create a call-site entry with no landing pad for the region between the
       // try-ranges.
       if (SawPotentiallyThrowing && Asm->MAI->usesCFIForEH()) {
         CallSiteEntry Site = { LastLabel, BeginLabel, nullptr, 0 };
         CallSites.push_back(Site);
         PreviousIsInvoke = false;
+        if (!IsInCold)
+          RegularCallCount++;
       }
 
       LastLabel = LandingPad->EndLabels[P.RangeIndex];
       assert(BeginLabel && LastLabel && "Invalid landing pad!");
 
       if (!LandingPad->LandingPadLabel) {
         // Create a gap.
         PreviousIsInvoke = false;
for (const auto &MI : MBB) {
           if (Site.LPad == Prev.LPad && Site.Action == Prev.Action) {
             // Extend the range of the previous entry.
             Prev.EndLabel = Site.EndLabel;
             continue;
           }
         }
 
         // Otherwise, create a new call-site.
-        if (!IsSJLJ)
+        if (!IsSJLJ) {
           CallSites.push_back(Site);
-        else {
+          if (!IsInCold)
+            RegularCallCount++;
+        } else {
           // SjLj EH must maintain the call sites in the order assigned
           // to them by the SjLjPrepare pass.
           unsigned SiteNo = MMI->getCallSiteBeginLabel(BeginLabel);
           if (CallSites.size() < SiteNo)
             CallSites.resize(SiteNo);
           CallSites[SiteNo - 1] = Site;
         }
         PreviousIsInvoke = true;
       }
     }
   }
 
   // If some instruction between the previous try-range and the end of the
   // function may throw, create a call-site entry with no landing pad for the
   // region following the try-range.
   if (SawPotentiallyThrowing && !IsSJLJ && LastLabel != nullptr) {
     CallSiteEntry Site = { LastLabel, nullptr, nullptr, 0 };
     CallSites.push_back(Site);
+    if (!IsInCold)
+      RegularCallCount++;
   }
+
+  return RegularCallCount;
 }
 
 /// Emit landing pads and actions.
 ///
 /// The general organization of the table is complex, but the basic concepts are
 /// easy.  First there is a header which describes the location and organization
 /// of the three components that follow.
 ///
void EHStreamer::emitExceptionTable() {
   // landing pad site.
   SmallVector<ActionEntry, 32> Actions;
   SmallVector<unsigned, 64> FirstActions;
   unsigned SizeActions =
     computeActionsTable(LandingPads, Actions, FirstActions);
 
   // Compute the call-site table.
   SmallVector<CallSiteEntry, 64> CallSites;
-  computeCallSiteTable(CallSites, LandingPads, FirstActions);
+  unsigned RegularCallCount = computeCallSiteTable(CallSites, LandingPads,
+                                                   FirstActions);
 
   // Final tallies.
 
   // Call sites.
   bool IsSJLJ = Asm->MAI->getExceptionHandlingType() == ExceptionHandling::SjLj;
   bool HaveTTData = IsSJLJ ? (!TypeInfos.empty() || !FilterIds.empty()) : true;
 
-  unsigned CallSiteTableLength;
-  if (IsSJLJ)
-    CallSiteTableLength = 0;
-  else {
+  // Cold splitting
+  MCSymbol *EHFuncColdBeginSym = Asm->getFunctionColdBegin();
+  assert(!IsSJLJ || !EHFuncColdBeginSym && "Cold splitting is not supported with SJLJ exceptions");
+
+  unsigned CallSiteTableLength = 0, ColdCallSiteTableLength = 0;
+  if (!IsSJLJ) {
     unsigned SiteStartSize  = 4; // dwarf::DW_EH_PE_udata4
     unsigned SiteLengthSize = 4; // dwarf::DW_EH_PE_udata4
     unsigned LandingPadSize = 4; // dwarf::DW_EH_PE_udata4
-    CallSiteTableLength =
-      CallSites.size() * (SiteStartSize + SiteLengthSize + LandingPadSize);
+    unsigned CallSiteSize = SiteStartSize + SiteLengthSize + LandingPadSize;
+    CallSiteTableLength = RegularCallCount * CallSiteSize;
+    ColdCallSiteTableLength =
+      (CallSites.size() - RegularCallCount) * CallSiteSize;
   }
 
-  for (unsigned i = 0, e = CallSites.size(); i < e; ++i) {
+  for (unsigned i = 0, e = RegularCallCount; i < e; ++i) {
     CallSiteTableLength += getULEB128Size(CallSites[i].Action);
     if (IsSJLJ)
       CallSiteTableLength += getULEB128Size(i);
   }
 
+  for (unsigned i = RegularCallCount, e = CallSites.size(); i < e; ++i) {
+    ColdCallSiteTableLength += getULEB128Size(CallSites[i].Action);
+    if (IsSJLJ)
+      ColdCallSiteTableLength += getULEB128Size(i);
+  }
+
   // Type infos.
   MCSection *LSDASection = Asm->getObjFileLowering().getLSDASection();
   unsigned TTypeEncoding;
   unsigned TypeFormatSize;
 
   if (!HaveTTData) {
     // For SjLj exceptions, if there is no TypeInfo, then we just explicitly say
     // that we're omitting that bit.
void EHStreamer::emitExceptionTable() {
 
   // Emit the LSDA.
   MCSymbol *GCCETSym =
     Asm->OutContext.getOrCreateSymbol(Twine("GCC_except_table")+
                                       Twine(Asm->getFunctionNumber()));
   Asm->OutStreamer->EmitLabel(GCCETSym);
   Asm->OutStreamer->EmitLabel(Asm->getCurExceptionSym());
 
-  // Emit the LSDA header.
-  Asm->EmitEncodingByte(dwarf::DW_EH_PE_omit, "@LPStart");
-  Asm->EmitEncodingByte(TTypeEncoding, "@TType");
-
   // The type infos need to be aligned. GCC does this by inserting padding just
   // before the type infos. However, this changes the size of the exception
   // table, so you need to take this into account when you output the exception
   // table size. However, the size is output using a variable length encoding.
   // So by increasing the size by inserting padding, you may increase the number
   // of bytes used for writing the size. If it increases, say by one byte, then
   // you now need to output one less byte of padding to get the type infos
   // aligned. However this decreases the size of the exception table. This
   // changes the value you have to output for the exception table size. Due to
   // the variable length encoding, the number of bytes used for writing the
   // length may decrease. If so, you then have to increase the amount of
   // padding. And so on. If you look carefully at the GCC code you will see that
   // it indeed does this in a loop, going on and on until the values stabilize.
   // We chose another solution: don't output padding inside the table like GCC
   // does, instead output it before the table.
   unsigned SizeTypes = TypeInfos.size() * TypeFormatSize;
-  unsigned CallSiteTableLengthSize = getULEB128Size(CallSiteTableLength);
-  unsigned TTypeBaseOffset =
-    sizeof(int8_t) +                            // Call site format
-    CallSiteTableLengthSize +                   // Call site table length size
-    CallSiteTableLength +                       // Call site table length
-    SizeActions +                               // Actions size
-    SizeTypes;
-  unsigned TTypeBaseOffsetSize = getULEB128Size(TTypeBaseOffset);
-  unsigned TotalSize =
-    sizeof(int8_t) +                            // LPStart format
-    sizeof(int8_t) +                            // TType format
-    (HaveTTData ? TTypeBaseOffsetSize : 0) +    // TType base offset size
-    TTypeBaseOffset;                            // TType base offset
-  unsigned SizeAlign = (4 - TotalSize) & 3;
 
-  if (HaveTTData) {
-    // Account for any extra padding that will be added to the call site table
-    // length.
-    Asm->EmitULEB128(TTypeBaseOffset, "@TType base offset", SizeAlign);
-    SizeAlign = 0;
-  }
+  // Emit the LSDA header.
+  emitLSDAHeader(EHFuncColdBeginSym, ColdCallSiteTableLength,
+                 CallSiteTableLength, TTypeEncoding,
+                 SizeActions + SizeTypes);
 
   bool VerboseAsm = Asm->OutStreamer->isVerboseAsm();
 
   // SjLj Exception handling
   if (IsSJLJ) {
-    Asm->EmitEncodingByte(dwarf::DW_EH_PE_udata4, "Call site");
-
-    // Add extra padding if it wasn't added to the TType base offset.
-    Asm->EmitULEB128(CallSiteTableLength, "Call site table length", SizeAlign);
-
     // Emit the landing pad site information.
     unsigned idx = 0;
     for (SmallVectorImpl<CallSiteEntry>::const_iterator
          I = CallSites.begin(), E = CallSites.end(); I != E; ++I, ++idx) {
       const CallSiteEntry &S = *I;
 
       // Offset of the landing pad, counted in 16-byte bundles relative to the
       // @LPStart address.
if (IsSJLJ) {
     //
     //   * The position of the call-site.
     //   * The position of the landing pad.
     //   * The first action record for that call site.
     //
     // A missing entry in the call-site table indicates that a call is not
     // supposed to throw.
 
-    // Emit the landing pad call site table.
-    Asm->EmitEncodingByte(dwarf::DW_EH_PE_udata4, "Call site");
+    bool IsInCold = false;
 
-    // Add extra padding if it wasn't added to the TType base offset.
-    Asm->EmitULEB128(CallSiteTableLength, "Call site table length", SizeAlign);
+    //    Regular Fragment
+    //    ---------------------
+    //   | FunctionBegin:      |
+    //   |                     |
+    //   | ...                 |
+    //   |                     |
+    //   | FunctionEnd:        |
+    //    ---------------------
+    //
+    //       ..............
+    //
+    //    Cold Fragment
+    //    ---------------------
+    //   | FunctionColdBegin:  |
+    //   |                     |
+    //   | ...                 |
+    //   |                     |
+    //   | FunctionColdEnd:    |
+    //    ---------------------
+
+    MCSymbol *EHFuncBeginSym = Asm->getFunctionBegin();
+    MCSymbol *EHFuncEndSym = Asm->getFunctionEnd();
+    MCSymbol *PadBaseLabel = EHFuncColdBeginSym
+      ? EHFuncColdBeginSym
+      : EHFuncBeginSym;
 
     unsigned Entry = 0;
     for (SmallVectorImpl<CallSiteEntry>::const_iterator
          I = CallSites.begin(), E = CallSites.end(); I != E; ++I) {
       const CallSiteEntry &S = *I;
 
-      MCSymbol *EHFuncBeginSym = Asm->getFunctionBegin();
-
       MCSymbol *BeginLabel = S.BeginLabel;
-      if (!BeginLabel)
+      if (!BeginLabel) {
         BeginLabel = EHFuncBeginSym;
+      } else if (BeginLabel->getSection().getKind().isColdText()) {
+        IsInCold = true;
+        EHFuncBeginSym = EHFuncColdBeginSym;
+        EHFuncEndSym = Asm->getFunctionColdEnd();
+
+        // We emit a second LSDA header withing the first one, so we can
+        // reuse action and type tables.
+        Asm->OutStreamer->EmitLabel(Asm->getCurColdExceptionSym());
+        emitLSDAHeader(nullptr, 0, ColdCallSiteTableLength, TTypeEncoding,
+                       SizeActions + SizeTypes);
+      }
+
       MCSymbol *EndLabel = S.EndLabel;
       if (!EndLabel)
-        EndLabel = Asm->getFunctionEnd();
+        EndLabel = EHFuncEndSym;
 
       // Offset of the call site relative to the previous call site, counted in
       // number of 16-byte bundles. The first call site is counted relative to
       // the start of the procedure fragment.
       if (VerboseAsm)
         Asm->OutStreamer->AddComment(">> Call Site " + Twine(++Entry) + " <<");
       Asm->EmitLabelDifference(BeginLabel, EHFuncBeginSym, 4);
       if (VerboseAsm)
         Asm->OutStreamer->AddComment(Twine("  Call between ") +
                                      BeginLabel->getName() + " and " +
                                      EndLabel->getName());
       Asm->EmitLabelDifference(EndLabel, BeginLabel, 4);
 
       // Offset of the landing pad, counted in 16-byte bundles relative to the
       // @LPStart address.
       if (!S.LPad) {
         if (VerboseAsm)
           Asm->OutStreamer->AddComment("    has no landing pad");
         Asm->OutStreamer->EmitIntValue(0, 4/*size*/);
       } else {
         if (VerboseAsm)
           Asm->OutStreamer->AddComment(Twine("    jumps to ") +
                                        S.LPad->LandingPadLabel->getName());
-        Asm->EmitLabelDifference(S.LPad->LandingPadLabel, EHFuncBeginSym, 4);
+        Asm->EmitLabelDifference(S.LPad->LandingPadLabel, PadBaseLabel, 4);
       }
 
       // Offset of the first associated action record, relative to the start of
       // the action table. This value is biased by 1 (1 indicates the start of
       // the action table), and 0 indicates that there are no actions.
       if (VerboseAsm) {
         if (S.Action == 0)
           Asm->OutStreamer->AddComment("  On action: cleanup");
for (SmallVectorImpl<ActionEntry>::const_iterator
     Asm->EmitSLEB128(Action.NextAction);
   }
 
   emitTypeInfos(TTypeEncoding);
 
   Asm->EmitAlignment(2);
 }
 
+void EHStreamer::emitLSDAHeader(MCSymbol *LPStart, unsigned SubCallSiteTableLength,
+                                unsigned CallSiteTableLength, unsigned TTypeEncoding,
+                                unsigned SizeActionsAndTypes) {
+  unsigned ActionTableIndex = CallSiteTableLength;
+
+  unsigned AlignOffset =
+    sizeof(int8_t) +                            // LPStart format
+    sizeof(int8_t);                             // TType format
+
+  if (LPStart) {
+    // We we have a nested LSDA after the callsites, acount for it.
+    // In order to not duplicate Actions and types entries, we nest
+    // the cold LSDA header after the regular calls. This adjusts
+    // offsets and alignement to fit that pattern.
+    //
+    //   LSDA table layout:
+    //    ---------------------
+    //   | @LPStart Encoding   | <= Regular LSDA header
+    //   | @LPStart            |
+    //   | @Type Encoding      |
+    //   | @Type base offset   | <=|
+    //   | Call site encoding  |   |= Used to enfore cold LSDA header alignment
+    //   | Action table index  | <=|
+    //   |                     |
+    //   | Regular call sites  |
+    //   | ...                 |
+    //   |                     |
+    //   |                     |
+    //   | @LPStart Encoding   | <= Cold LSDA header
+    //   | @LPStart            |
+    //   | @Type Encoding      |
+    //   | @Type base offset   | <=|
+    //   | Call site encoding  |   |= Used to enfore type table alignement
+    //   | Action table index  | <=|
+    //   |                     |
+    //   | Cold call sites     |
+    //   | ...                 |
+    //   |                     |
+    //   |                     |
+    //   | Action table        | | Action table grows down
+    //   | ...                 | V
+    //   |                     |
+    //   |                     |
+    //   | ...                 | ^
+    //   | Type table          | | Type table grows up
+    //    ---------------------
+    //
+    if (SubCallSiteTableLength) {
+      unsigned TTypeSubHeaderOffset =
+        sizeof(int8_t) +                          // Call site format
+        getULEB128Size(SubCallSiteTableLength);   // Call site table length
+
+      unsigned TTypeSubBodyOffset =
+        SubCallSiteTableLength +                  // Call site table
+        SizeActionsAndTypes;                      // Actions and Types size
+
+      unsigned SubLSDAHeaderSize =
+        sizeof(int8_t) +                          // LPStart format
+        sizeof(int8_t) +                          // TType format
+        TTypeSubHeaderOffset;
+
+      if (TTypeEncoding != dwarf::DW_EH_PE_omit)
+        SubLSDAHeaderSize += getULEB128Size(TTypeSubHeaderOffset + TTypeSubBodyOffset);
+
+      // Make sure it is aligned.
+      unsigned SubLSDAAlignOffet = SubLSDAHeaderSize + TTypeSubBodyOffset;
+      SubLSDAHeaderSize += ((4 - SubLSDAAlignOffet) & 3);
+
+      ActionTableIndex += SubLSDAHeaderSize + SubCallSiteTableLength;
+
+      // We make sure nested LSDA is aligned properly
+      AlignOffset -= ActionTableIndex;
+      AlignOffset -= SizeActionsAndTypes;
+      AlignOffset += CallSiteTableLength;
+    }
+
+    auto PtrSize = Asm->getDataLayout().getPointerSize();
+
+    // LPStart Pointer
+    AlignOffset += PtrSize;
+
+    // If there is a cold fragment, landing pads are in there.
+    Asm->EmitEncodingByte(dwarf::DW_EH_PE_absptr, "@LPStart");
+    Asm->OutStreamer->EmitSymbolValue(LPStart, PtrSize);
+  } else {
+    assert(!SubCallSiteTableLength && "Nested LDSA require explicit LPStart");
+    Asm->EmitEncodingByte(dwarf::DW_EH_PE_omit, "@LPStart");
+  }
+
+  unsigned TTypeBaseOffset =
+    sizeof(int8_t) +                            // Call site format
+    getULEB128Size(ActionTableIndex) +          // Action table index size
+    ActionTableIndex +                          // Action table index
+    SizeActionsAndTypes;                        // Actions and Types size
+
+  AlignOffset += TTypeBaseOffset;               // TType base offset
+
+  Asm->EmitEncodingByte(TTypeEncoding, "@TType");
+  if (TTypeEncoding != dwarf::DW_EH_PE_omit) {
+    AlignOffset += getULEB128Size(TTypeBaseOffset);
+
+    unsigned SizeAlign = (4 - AlignOffset) & 3;
+    AlignOffset += SizeAlign;
+
+    // Account for any extra padding that will be added to the call site table
+    // length.
+    Asm->EmitULEB128(TTypeBaseOffset, "@TType base offset", SizeAlign);
+  }
+
+  // Emit the landing pad call site table.
+  Asm->EmitEncodingByte(dwarf::DW_EH_PE_udata4, "Call site");
+
+  // Add extra padding if it wasn't added to the TType base offset.
+  Asm->EmitULEB128(ActionTableIndex, "Action table index", (4 - AlignOffset) & 3);
+}
+
 void EHStreamer::emitTypeInfos(unsigned TTypeEncoding) {
   const std::vector<const GlobalValue *> &TypeInfos = MMI->getTypeInfos();
   const std::vector<unsigned> &FilterIds = MMI->getFilterIds();
 
   bool VerboseAsm = Asm->OutStreamer->isVerboseAsm();
 
   int Entry = 0;
   // Emit the Catch TypeInfos.

lib/MC/MCObjectFileInfo.cpp

void MCObjectFileInfo::initMachOMCObjectFileInfo(Triple T) {
   // .comm doesn't support alignment before Leopard.
   if (T.isMacOSX() && T.isMacOSXVersionLT(10, 5))
     CommDirectiveSupportsAlignment = false;
 
   TextSection // .text
     = Ctx->getMachOSection("__TEXT", "__text",
                            MachO::S_ATTR_PURE_INSTRUCTIONS,
                            SectionKind::getText());
+  ColdTextSection // .text.cold
+    = Ctx->getMachOSection("__TEXT", "__text.cold",
+                           MachO::S_ATTR_PURE_INSTRUCTIONS,
+                           SectionKind::getColdText());
   DataSection // .data
       = Ctx->getMachOSection("__DATA", "__data", 0, SectionKind::getData());
 
   // BSSSection might not be expected initialized on msvc.
   BSSSection = nullptr;
 
   TLSDataSection // .tdata
       = Ctx->getMachOSection("__DATA", "__thread_data",
void MCObjectFileInfo::initELFMCObjectFileInfo(Triple T) {
 
   // ELF
   BSSSection = Ctx->getELFSection(".bss", ELF::SHT_NOBITS,
                                   ELF::SHF_WRITE | ELF::SHF_ALLOC);
 
   TextSection = Ctx->getELFSection(".text", ELF::SHT_PROGBITS,
                                    ELF::SHF_EXECINSTR | ELF::SHF_ALLOC);
 
+  ColdTextSection = Ctx->getELFSection(".text.cold", ELF::SHT_PROGBITS,
+                                       ELF::SHF_EXECINSTR | ELF::SHF_ALLOC);
+
   DataSection = Ctx->getELFSection(".data", ELF::SHT_PROGBITS,
                                    ELF::SHF_WRITE | ELF::SHF_ALLOC);
 
   ReadOnlySection =
       Ctx->getELFSection(".rodata", ELF::SHT_PROGBITS, ELF::SHF_ALLOC);
 
   TLSDataSection =
       Ctx->getELFSection(".tdata", ELF::SHT_PROGBITS,
void MCObjectFileInfo::InitMCObjectFileInfo(const Triple &TheTriple,
   OmitDwarfIfHaveCompactUnwind = false;
 
   PersonalityEncoding = LSDAEncoding = FDECFIEncoding = TTypeEncoding =
       dwarf::DW_EH_PE_absptr;
 
   CompactUnwindDwarfEHFrameOnly = 0;
 
   EHFrameSection = nullptr;             // Created on demand.
+  ColdTextSection = nullptr;            // Only work on some plateforms.
   CompactUnwindSection = nullptr;       // Used only by selected targets.
   DwarfAccelNamesSection = nullptr;     // Used only by selected targets.
   DwarfAccelObjCSection = nullptr;      // Used only by selected targets.
   DwarfAccelNamespaceSection = nullptr; // Used only by selected targets.
   DwarfAccelTypesSection = nullptr;     // Used only by selected targets.
 
   TT = TheTriple;
 

lib/MC/MCStreamer.cpp

 }
 
 MCDwarfFrameInfo *MCStreamer::getCurrentDwarfFrameInfo() {
   if (DwarfFrameInfos.empty())
     return nullptr;
   return &DwarfFrameInfos.back();
 }
 
+bool MCStreamer::hasUnfinishedDwarfFrameInfo() {
+  MCDwarfFrameInfo *CurFrame = getCurrentDwarfFrameInfo();
+  return CurFrame && !CurFrame->End;
+}
+
+MCDwarfFrameInfo MCStreamer::popDwarfFrameInfo() {
+  assert(hasUnfinishedDwarfFrameInfo() && "Can only pop unfinished frames");
+  auto DFI = DwarfFrameInfos.back();
+  DwarfFrameInfos.pop_back();
+  return DFI;
+}
+
+void MCStreamer::pushDwarfFrameInfo(MCDwarfFrameInfo DFI) {
+  assert(!hasUnfinishedDwarfFrameInfo() && "Pushing a frame before finishing the previous one!");
+  DwarfFrameInfos.push_back(DFI);
+}
+
 void MCStreamer::EnsureValidDwarfFrame() {
   MCDwarfFrameInfo *CurFrame = getCurrentDwarfFrameInfo();
   if (!CurFrame || CurFrame->End)
     report_fatal_error("No open frame");
 }
 
 unsigned MCStreamer::EmitCVFileDirective(unsigned FileNo, StringRef Filename) {
   return getContext().getCVFile(Filename, FileNo);
if (TS)
     TS->emitLabel(Symbol);
 }
 
 void MCStreamer::EmitCFISections(bool EH, bool Debug) {
   assert(EH || Debug);
 }
 
 void MCStreamer::EmitCFIStartProc(bool IsSimple) {
-  MCDwarfFrameInfo *CurFrame = getCurrentDwarfFrameInfo();
-  if (CurFrame && !CurFrame->End)
-    report_fatal_error("Starting a frame before finishing the previous one!");
+  assert(!hasUnfinishedDwarfFrameInfo() && "Starting a frame before finishing the previous one!");
 
   MCDwarfFrameInfo Frame;
   Frame.IsSimple = IsSimple;
   EmitCFIStartProcImpl(Frame);
 
   const MCAsmInfo* MAI = Context.getAsmInfo();
   if (MAI) {
     for (const MCCFIInstruction& Inst : MAI->getInitialFrameState()) {

lib/MC/MCTargetOptions.cpp

 
 namespace llvm {
 
 MCTargetOptions::MCTargetOptions()
     : SanitizeAddress(false), MCRelaxAll(false), MCNoExecStack(false),
       MCFatalWarnings(false), MCNoWarn(false), MCSaveTempLabels(false),
       MCUseDwarfDirectory(false), MCIncrementalLinkerCompatible(false),
       ShowMCEncoding(false), ShowMCInst(false), AsmVerbose(false),
-      DwarfVersion(0), ABIName() {}
+      SplitColdCode(false), DwarfVersion(0), ABIName() {}
 
 StringRef MCTargetOptions::getABIName() const {
   return ABIName;
 }
 
 } // end namespace llvm

test/CodeGen/X86/coldsplit.ll

+; RUN: llc -split-cold-code < %s | FileCheck %s
+
+target triple = "x86_64-apple-macosx10.11.0"
+
+declare i32 @foo();
+
+declare i32 @bar();
+
+define i32 @branchweightcoldsplit(i32 %a) {
+  %1 = icmp sgt i32 %a, 1
+  br i1 %1, label %2, label %4, !prof !0
+
+; <label>:2:                                      ; preds = %0
+  %3 = call i32 @foo()
+  br label %6
+
+; <label>:4:                                      ; preds = %0
+  %5 = call i32 @bar()
+  br label %6
+
+; <label>:6:                                      ; preds = %4, %2
+  %.0 = phi i32 [ %3, %2 ], [ %5, %4 ]
+  ret i32 %.0
+}
+
+!0 = !{!"branch_weights", i32 65536, i32 0}
+
+; CHECK-LABEL: _branchweightcoldsplit: ## @branchweightcoldsplit
+; CHECK: callq _foo
+; CHECK: retq
+; CHECK: .section __TEXT,__text.cold,regular,pure_instructions
+; CHECK-LABEL: _branchweightcoldsplit$cold
+; CHECK: callq _bar
+; CHECK: retq
+
+declare i32 @pers(...)
+
+define i32 @cleanup() personality i32 (...)* @pers {
+  %1 = invoke i32 @foo()
+          to label %2 unwind label %3
+
+; <label>:2:                                      ; preds = %0
+  ret i32 0
+
+; <label>:3:                                      ; preds = %0
+  %lp = landingpad { i8*, i32 }
+          cleanup
+  %4 = call i32 @bar()
+  ret i32 %4
+}
+
+; CHECK-LABEL: _cleanup: ## @cleanup
+; CHECK: .cfi_startproc
+; CHECK: .cfi_personality 155, _pers
+; CHECK: .cfi_lsda 16, Lexception0
+; CHECK: callq _foo
+; CHECK: retq
+; CHECK: .section __TEXT,__text.cold,regular,pure_instructions
+; CHECK-LABEL: _cleanup$cold
+; CHECK: .cfi_startproc
+; CHECK: .cfi_personality 155, _pers
+; CHECK: .cfi_lsda 16, Lcold_exception0
+; CHECK: callq _bar
+; CHECK: retq