[BOLT] Support passing fragments to code emission

[FPar]

Summary

This changes code emission such that it can emit specific function
fragments instead of scanning all basic blocks of a function and just
emitting those that are hot or cold.

To implement this, FunctionLayout explicitly distinguishes the "main"
fragment (i.e. the one that contains the entry block and is associated
with the original symbol) from "split" fragments. Additionally,
BinaryFunction receives support for multiple cold symbols - one for
each split fragment.

[FPar]

Clean diff up

[FPar]

Fix cold fragment size calculation

[FPar]

Add assertion messages

[FPar]

Fix fragment size calculation

Emitting label before switching section led to undefined values.

[FPar]

Support multiple fragments for addExtraSymbols

[FPar]

Rebase on earlier commit

[rafauler]

There's some weird handling of fragment images, but I guess the idea is to implement them in future diffs - maybe https://reviews.llvm.org/D130521 ? We need to coordinate this well so we don't lose track of everything that needs to be updated after this diff lands. I have some comments in the parts of the code that are reading a bit odd to me at the moment.

bolt/lib/Core/BinaryEmitter.cpp

387

I guess the fact that we don't have individual end labels for each fragment is going to be a problem for determining the correct size of these symbols in emitELFSize. But according to 247b4181a306aff348c1d2d8c3847280866a7f32, this symbol table isn't terribly important. But I would rather not have incorrect code.

bolt/lib/Rewrite/RewriteInstance.cpp

3190–3191

Move outside of the loop / under "if Function->isSplit"?

3729–3760

Emitting the same coldPart over and over for each fragment?

4530–4531

Same image for all fragments? I guess this will be implemented in a follow-up diff? ps: I don't see that implemented in https://reviews.llvm.org/D130521

[rafauler]

bolt/lib/Core/BinaryEmitter.cpp

387

I see this is implemented in https://reviews.llvm.org/D130520 now.

[FPar]

Update ColdCodeSectionName after loop

[FPar]

I am aware that the ordering of diffs not ideal. The diffs D130052, D130520 and D130521 are the baseline of changes required to get simple programs running. I tried to group related changes, but they all depend on each other in some way.

bolt/lib/Core/BinaryEmitter.cpp

387

Correct.

bolt/lib/Rewrite/RewriteInstance.cpp

3190–3191

Moved to getLayout().isSplit(), because Function->isSplit() is a slightly different check.

3729–3760

This is addressed in D132051. This diff only takes care that ColdSection has the right output address.

4530–4531

Yes. Again, D132051.

[rafauler]

LGTM

bolt/include/bolt/Core/BinaryEmitter.h

 #include "llvm/ADT/StringRef.h"
 
 namespace llvm {
 class MCStreamer;
 
 namespace bolt {
 class BinaryContext;
 class BinaryFunction;
+class FunctionFragment;
 
 /// Emit all code and data from the BinaryContext \p BC into the \p Streamer.
 ///
 /// \p OrgSecPrefix is used to modify name of emitted original sections
 /// contained in \p BC. This is done to distinguish them from sections emitted
 /// by LLVM backend.
 void emitBinaryContext(MCStreamer &Streamer, BinaryContext &BC,
                        StringRef OrgSecPrefix = "");
 
 /// Emit \p BF function code. The caller is responsible for emitting function
 /// symbol(s) and setting the section to emit the code to.
 void emitFunctionBody(MCStreamer &Streamer, BinaryFunction &BF,
-                      bool EmitColdPart, bool EmitCodeOnly = false);
+                      const FunctionFragment &FF, bool EmitCodeOnly);
 
 } // namespace bolt
 } // namespace llvm
 
 #endif

bolt/include/bolt/Core/BinaryFunction.h

private:
   struct CompareBasicBlockOffsets {
     bool operator()(const BasicBlockOffset &A,
                     const BasicBlockOffset &B) const {
       return A.first < B.first;
     }
   };
   SmallVector<BasicBlockOffset, 0> BasicBlockOffsets;
 
-  MCSymbol *ColdSymbol{nullptr};
+  SmallVector<MCSymbol *, 0> ColdSymbols;
 
   /// Symbol at the end of the function.
   mutable MCSymbol *FunctionEndLabel{nullptr};
 
   /// Symbol at the end of the cold part of split function.
   mutable MCSymbol *FunctionColdEndLabel{nullptr};
 
   /// Unique number associated with the function.
uint32_t getNumNonPseudos() const {
     uint32_t N = 0;
     for (const BinaryBasicBlock &BB : blocks())
       N += BB.getNumNonPseudos();
     return N;
   }
 
   /// Return MC symbol associated with the function.
   /// All references to the function should use this symbol.
-  MCSymbol *getSymbol() { return Symbols[0]; }
+  MCSymbol *getSymbol(const FragmentNum Fragment = FragmentNum::hot()) {
+    if (Fragment == FragmentNum::hot())
+      return Symbols[0];
+
+    size_t ColdSymbolIndex = Fragment.get() - 1;
+    if (ColdSymbolIndex >= ColdSymbols.size())
+      ColdSymbols.resize(ColdSymbolIndex + 1);
+
+    MCSymbol *&ColdSymbol = ColdSymbols[ColdSymbolIndex];
+    if (ColdSymbol == nullptr) {
+      SmallString<10> Appendix = formatv(".cold.{0}", ColdSymbolIndex);
+      ColdSymbol = BC.Ctx->getOrCreateSymbol(
+          NameResolver::append(Symbols[0]->getName(), Appendix));
+    }
+
+    return ColdSymbol;
+  }
 
   /// Return MC symbol associated with the function (const version).
   /// All references to the function should use this symbol.
   const MCSymbol *getSymbol() const { return Symbols[0]; }
 
   /// Return a list of symbols associated with the main entry of the function.
   SymbolListTy &getSymbols() { return Symbols; }
   const SymbolListTy &getSymbols() const { return Symbols; }
public:
   /// Stop calling the function after false is returned by the callback.
   ///
   /// Pass an offset of the entry point in the input binary and a corresponding
   /// global symbol to the callback function.
   ///
   /// Return true of all callbacks returned true, false otherwise.
   bool forEachEntryPoint(EntryPointCallbackTy Callback) const;
 
-  MCSymbol *getColdSymbol() {
-    if (ColdSymbol)
-      return ColdSymbol;
-
-    ColdSymbol = BC.Ctx->getOrCreateSymbol(
-        NameResolver::append(getSymbol()->getName(), ".cold.0"));
-
-    return ColdSymbol;
-  }
-
   /// Return MC symbol associated with the end of the function.
   MCSymbol *getFunctionEndLabel() const {
     assert(BC.Ctx && "cannot be called with empty context");
     if (!FunctionEndLabel) {
       std::unique_lock<std::shared_timed_mutex> Lock(BC.CtxMutex);
       FunctionEndLabel = BC.Ctx->createNamedTempSymbol("func_end");
     }
     return FunctionEndLabel;
public:
   }
 
   /// Get output code section.
   ErrorOr<BinarySection &> getCodeSection() const {
     return BC.getUniqueSectionByName(getCodeSectionName());
   }
 
   /// Return cold code section name for the function.
-  StringRef getColdCodeSectionName() const {
-    return StringRef(ColdCodeSectionName);
+  std::string getColdCodeSectionName(const FragmentNum Fragment) const {
+    std::string Result = ColdCodeSectionName;
+    if (Fragment != FragmentNum::cold())
+      Result.append(".").append(std::to_string(Fragment.get() - 1));
+    return Result;
   }
 
   /// Assign a section name for the cold part of the function.
   void setColdCodeSectionName(StringRef Name) {
     ColdCodeSectionName = std::string(Name);
   }
 
   /// Get output code section for cold code of this function.
-  ErrorOr<BinarySection &> getColdCodeSection() const {
-    return BC.getUniqueSectionByName(getColdCodeSectionName());
+  ErrorOr<BinarySection &>
+  getColdCodeSection(const FragmentNum Fragment) const {
+    return BC.getUniqueSectionByName(getColdCodeSectionName(Fragment));
   }
 
   /// Return true iif the function will halt execution on entry.
   bool trapsOnEntry() const { return TrapsOnEntry; }
 
   /// Make the function always trap on entry. Other than the trap instruction,
   /// the function body will be empty.
   void setTrapOnEntry();

bolt/include/bolt/Core/FunctionLayout.h

 private:
   FragmentNum Num;
   const FunctionLayout &Layout;
 
   FunctionFragment(FragmentNum Num, const FunctionLayout &Layout)
       : Num(Num), Layout(Layout) {}
 
 public:
+  FragmentNum getFragmentNum() const { return Num; }
+  bool isMainFragment() const { return Num.get() == 0; }
+  bool isSplitFragment() const { return Num.get() > 0; }
+
   unsigned size() const;
   bool empty() const;
   const_iterator begin() const;
   const_iterator end() const;
   BinaryBasicBlock *front() const;
 
   friend class FunctionLayout;
   friend class FragmentIterator;

 
 public:
   /// Add an empty fragment.
   FunctionFragment addFragment();
 
   /// Return the fragment identified by Num.
   FunctionFragment getFragment(FragmentNum Num) const;
 
+  /// Get the fragment that contains all entry blocks and other blocks that
+  /// cannot be split.
+  FunctionFragment getMainFragment() const {
+    return getFragment(FragmentNum::hot());
+  }
+
+  /// Get the fragment that contains all entry blocks and other blocks that
+  /// cannot be split.
+  iterator_range<const_iterator> getSplitFragments() const {
+    return {++fragment_begin(), fragment_end()};
+  }
+
   /// Find the fragment that contains BB.
   FunctionFragment findFragment(const BinaryBasicBlock *BB) const;
 
   /// Add BB to the end of the last fragment.
   void addBasicBlock(BinaryBasicBlock *BB);
 
   /// Insert range of basic blocks after InsertAfter. If InsertAfter is nullptr,
   /// the blocks will be inserted at the start of the function.

bolt/lib/Core/BinaryContext.cpp

 #include "llvm/MC/MCSubtargetInfo.h"
 #include "llvm/MC/MCSymbol.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/Regex.h"
 #include <algorithm>
 #include <functional>
 #include <iterator>
+#include <numeric>
 #include <unordered_set>
 
 using namespace llvm;
 
 #undef  DEBUG_TYPE
 #define DEBUG_TYPE "bolt"
 
 namespace opts {
BinaryContext::calculateEmittedSize(BinaryFunction &BF, bool FixBranches) {
   Streamer->initSections(false, *STI);
 
   MCSection *Section = MCEInstance.LocalMOFI->getTextSection();
   Section->setHasInstructions(true);
 
   // Create symbols in the LocalCtx so that they get destroyed with it.
   MCSymbol *StartLabel = LocalCtx->createTempSymbol();
   MCSymbol *EndLabel = LocalCtx->createTempSymbol();
-  MCSymbol *ColdStartLabel = LocalCtx->createTempSymbol();
-  MCSymbol *ColdEndLabel = LocalCtx->createTempSymbol();
 
   Streamer->switchSection(Section);
   Streamer->emitLabel(StartLabel);
-  emitFunctionBody(*Streamer, BF, /*EmitColdPart=*/false,
+  emitFunctionBody(*Streamer, BF, BF.getLayout().getMainFragment(),
                    /*EmitCodeOnly=*/true);
   Streamer->emitLabel(EndLabel);
 
-  if (BF.isSplit()) {
-    MCSectionELF *ColdSection =
-        LocalCtx->getELFSection(BF.getColdCodeSectionName(), ELF::SHT_PROGBITS,
-                                ELF::SHF_EXECINSTR | ELF::SHF_ALLOC);
-    ColdSection->setHasInstructions(true);
+  using LabelRange = std::pair<const MCSymbol *, const MCSymbol *>;
+  SmallVector<LabelRange> SplitLabels;
+  for (const FunctionFragment FF : BF.getLayout().getSplitFragments()) {
+    MCSymbol *const SplitStartLabel = LocalCtx->createTempSymbol();
+    MCSymbol *const SplitEndLabel = LocalCtx->createTempSymbol();
+    SplitLabels.emplace_back(SplitStartLabel, SplitEndLabel);
+
+    MCSectionELF *const SplitSection = LocalCtx->getELFSection(
+        BF.getColdCodeSectionName(FF.getFragmentNum()), ELF::SHT_PROGBITS,
+    SplitSection->setHasInstructions(true);
+    Streamer->switchSection(SplitSection);
 
-    Streamer->switchSection(ColdSection);
-    Streamer->emitLabel(ColdStartLabel);
-    emitFunctionBody(*Streamer, BF, /*EmitColdPart=*/true,
-                     /*EmitCodeOnly=*/true);
-    Streamer->emitLabel(ColdEndLabel);
-    // To avoid calling MCObjectStreamer::flushPendingLabels() which is private
+    Streamer->emitLabel(SplitStartLabel);
+    emitFunctionBody(*Streamer, BF, FF, /*EmitCodeOnly=*/true);
+    Streamer->emitLabel(SplitEndLabel);
+    // To avoid calling MCObjectStreamer::flushPendingLabels() which is
+    // private
     Streamer->emitBytes(StringRef(""));
     Streamer->switchSection(Section);
   }
 
   // To avoid calling MCObjectStreamer::flushPendingLabels() which is private or
   // MCStreamer::Finish(), which does more than we want
   Streamer->emitBytes(StringRef(""));
 
   MCAssembler &Assembler =
       static_cast<MCObjectStreamer *>(Streamer.get())->getAssembler();
   MCAsmLayout Layout(Assembler);
   Assembler.layout(Layout);
 
   const uint64_t HotSize =
       Layout.getSymbolOffset(*EndLabel) - Layout.getSymbolOffset(*StartLabel);
-  const uint64_t ColdSize = BF.isSplit()
-                                ? Layout.getSymbolOffset(*ColdEndLabel) -
-                                      Layout.getSymbolOffset(*ColdStartLabel)
-                                : 0ULL;
+  const uint64_t ColdSize =
+      std::accumulate(SplitLabels.begin(), SplitLabels.end(), 0ULL,
+                      [&](const uint64_t Accu, const LabelRange &Labels) {
+                        return Accu + Layout.getSymbolOffset(*Labels.second) -
+                               Layout.getSymbolOffset(*Labels.first);
+                      });
 
   // Clean-up the effect of the code emission.
   for (const MCSymbol &Symbol : Assembler.symbols()) {
     MCSymbol *MutableSymbol = const_cast<MCSymbol *>(&Symbol);
     MutableSymbol->setUndefined();
     MutableSymbol->setIsRegistered(false);
   }
 

bolt/lib/Core/BinaryEmitter.cpp

public:
   BinaryEmitter(MCStreamer &Streamer, BinaryContext &BC)
       : Streamer(Streamer), BC(BC) {}
 
   /// Emit all code and data.
   void emitAll(StringRef OrgSecPrefix);
 
   /// Emit function code. The caller is responsible for emitting function
   /// symbol(s) and setting the section to emit the code to.
-  void emitFunctionBody(BinaryFunction &BF, bool EmitColdPart,
+  void emitFunctionBody(BinaryFunction &BF, const FunctionFragment &FF,
                         bool EmitCodeOnly = false);
 
 private:
   /// Emit function code.
   void emitFunctions();
 
   /// Emit a single function.
-  bool emitFunction(BinaryFunction &BF, bool EmitColdPart);
+  bool emitFunction(BinaryFunction &BF, const FunctionFragment &FF);
 
   /// Helper for emitFunctionBody to write data inside a function
   /// (used for AArch64)
   void emitConstantIslands(BinaryFunction &BF, bool EmitColdPart,
                            BinaryFunction *OnBehalfOf = nullptr);
 
   /// Emit jump tables for the function.
   void emitJumpTables(const BinaryFunction &BF);
for (BinaryFunction *Function : Functions) {
       // Was any part of the function emitted.
       bool Emitted = false;
 
       // Turn off Intel JCC Erratum mitigation for cold code if requested
       if (HasProfile && opts::X86AlignBranchBoundaryHotOnly &&
           !Function->hasValidProfile())
         Streamer.setAllowAutoPadding(false);
 
-      Emitted |= emitFunction(*Function, /*EmitColdPart=*/false);
+      const FunctionLayout &Layout = Function->getLayout();
+      Emitted |= emitFunction(*Function, Layout.getMainFragment());
 
       if (Function->isSplit()) {
         if (opts::X86AlignBranchBoundaryHotOnly)
           Streamer.setAllowAutoPadding(false);
-        Emitted |= emitFunction(*Function, /*EmitColdPart=*/true);
+
+        assert((Layout.fragment_size() == 1 || Function->isSimple()) &&
+               "Only simple functions can have fragments");
+        for (const FunctionFragment FF : Layout.getSplitFragments()) {
+          // Skip empty fragments so no symbols and sections for empty fragments
+          // are generated
+          if (FF.empty() && !Function->hasConstantIsland())
+            continue;
+          Emitted |= emitFunction(*Function, FF);
         }
+      }
+
       Streamer.setAllowAutoPadding(OriginalAllowAutoPadding);
 
       if (Emitted)
         Function->setEmitted(/*KeepCFG=*/opts::PrintCacheMetrics);
     }
   };
 
   // Mark the start of hot text.
void BinaryEmitter::emitFunctions() {
 
   // Mark the end of hot text.
   if (opts::HotText) {
     Streamer.switchSection(BC.getTextSection());
     Streamer.emitLabel(BC.getHotTextEndSymbol());
   }
 }
 
-bool BinaryEmitter::emitFunction(BinaryFunction &Function, bool EmitColdPart) {
+bool BinaryEmitter::emitFunction(BinaryFunction &Function,
+                                 const FunctionFragment &FF) {
   if (Function.size() == 0 && !Function.hasIslandsInfo())
     return false;
 
   if (Function.getState() == BinaryFunction::State::Empty)
     return false;
 
-  MCSection *Section =
-      BC.getCodeSection(EmitColdPart ? Function.getColdCodeSectionName()
-                                     : Function.getCodeSectionName());
+  MCSection *Section = BC.getCodeSection(
+      FF.isSplitFragment()
+          ? Function.getColdCodeSectionName(FF.getFragmentNum())
   Streamer.switchSection(Section);
   Section->setHasInstructions(true);
   BC.Ctx->addGenDwarfSection(Section);
 
   if (BC.HasRelocations) {
     // Set section alignment to at least maximum possible object alignment.
     // We need this to support LongJmp and other passes that calculates
     // tentative layout.
     if (Section->getAlignment() < opts::AlignFunctions)
       Section->setAlignment(Align(opts::AlignFunctions));
 
     Streamer.emitCodeAlignment(BinaryFunction::MinAlign, &*BC.STI);
-    uint16_t MaxAlignBytes = EmitColdPart ? Function.getMaxColdAlignmentBytes()
-                                          : Function.getMaxAlignmentBytes();
+    uint16_t MaxAlignBytes = FF.isSplitFragment()
+                                 ? Function.getMaxColdAlignmentBytes()
     if (MaxAlignBytes > 0)
       Streamer.emitCodeAlignment(Function.getAlignment(), &*BC.STI,
                                  MaxAlignBytes);
   } else {
     Streamer.emitCodeAlignment(Function.getAlignment(), &*BC.STI);
   }
 
   MCContext &Context = Streamer.getContext();
   const MCAsmInfo *MAI = Context.getAsmInfo();
 
-  MCSymbol *StartSymbol = nullptr;
+  MCSymbol *const StartSymbol = Function.getSymbol(FF.getFragmentNum());
 
   // Emit all symbols associated with the main function entry.
-  if (!EmitColdPart) {
-    StartSymbol = Function.getSymbol();
+  if (FF.isMainFragment()) {
     for (MCSymbol *Symbol : Function.getSymbols()) {
       Streamer.emitSymbolAttribute(Symbol, MCSA_ELF_TypeFunction);
       Streamer.emitLabel(Symbol);
     }
   } else {
-    StartSymbol = Function.getColdSymbol();
     Streamer.emitSymbolAttribute(StartSymbol, MCSA_ELF_TypeFunction);
     Streamer.emitLabel(StartSymbol);
   }
 
   // Emit CFI start
   if (Function.hasCFI()) {
+    assert(Function.getLayout().isHotColdSplit() &&
+           "Exceptions supported only with hot/cold splitting.");
     Streamer.emitCFIStartProc(/*IsSimple=*/false);
     if (Function.getPersonalityFunction() != nullptr)
       Streamer.emitCFIPersonality(Function.getPersonalityFunction(),
                                   Function.getPersonalityEncoding());
-    MCSymbol *LSDASymbol =
-        EmitColdPart ? Function.getColdLSDASymbol() : Function.getLSDASymbol();
+    MCSymbol *LSDASymbol = FF.isSplitFragment() ? Function.getColdLSDASymbol()
+                                                : Function.getLSDASymbol();
     if (LSDASymbol)
       Streamer.emitCFILsda(LSDASymbol, BC.LSDAEncoding);
     else
       Streamer.emitCFILsda(0, dwarf::DW_EH_PE_omit);
     // Emit CFI instructions relative to the CIE
     for (const MCCFIInstruction &CFIInstr : Function.cie()) {
       // Only write CIE CFI insns that LLVM will not already emit
       const std::vector<MCCFIInstruction> &FrameInstrs =
for (std::string &Name : opts::BreakFunctionNames) {
       if (Function.hasNameRegex(Name)) {
         Streamer.emitIntValue(0x0B0F, 2); // UD2: 0F 0B
         break;
       }
     }
   }
 
   // Emit code.
-  emitFunctionBody(Function, EmitColdPart, /*EmitCodeOnly=*/false);
+  emitFunctionBody(Function, FF, /*EmitCodeOnly=*/false);
 
   // Emit padding if requested.
   if (size_t Padding = opts::padFunction(Function)) {
     LLVM_DEBUG(dbgs() << "BOLT-DEBUG: padding function " << Function << " with "
                       << Padding << " bytes\n");
     Streamer.emitFill(Padding, MAI->getTextAlignFillValue());
   }
 
   if (opts::MarkFuncs)
     Streamer.emitIntValue(BC.MIB->getTrapFillValue(), 1);
 
   // Emit CFI end
   if (Function.hasCFI())
     Streamer.emitCFIEndProc();
 
-  MCSymbol *EndSymbol = EmitColdPart ? Function.getFunctionColdEndLabel()
-                                     : Function.getFunctionEndLabel();
+  MCSymbol *EndSymbol = FF.isSplitFragment()
+                            ? Function.getFunctionColdEndLabel()
   Streamer.emitLabel(EndSymbol);
 
   if (MAI->hasDotTypeDotSizeDirective()) {
     const MCExpr *SizeExpr = MCBinaryExpr::createSub(
         MCSymbolRefExpr::create(EndSymbol, Context),
         MCSymbolRefExpr::create(StartSymbol, Context), Context);
     Streamer.emitELFSize(StartSymbol, SizeExpr);
   }
 
   if (opts::UpdateDebugSections && Function.getDWARFUnit())
     emitLineInfoEnd(Function, EndSymbol);
 
   // Exception handling info for the function.
-  emitLSDA(Function, EmitColdPart);
+  emitLSDA(Function, FF.isSplitFragment());
 
-  if (!EmitColdPart && opts::JumpTables > JTS_NONE)
+  if (FF.isMainFragment() && opts::JumpTables > JTS_NONE)
     emitJumpTables(Function);
 
   return true;
 }
 
-void BinaryEmitter::emitFunctionBody(BinaryFunction &BF, bool EmitColdPart,
+void BinaryEmitter::emitFunctionBody(BinaryFunction &BF,
+                                     const FunctionFragment &FF,
                                      bool EmitCodeOnly) {
-  if (!EmitCodeOnly && EmitColdPart && BF.hasConstantIsland())
+  if (!EmitCodeOnly && FF.isSplitFragment() && BF.hasConstantIsland()) {
+    assert(FF.getFragmentNum() == FragmentNum::cold() &&
+           "Constant island support only with hot/cold split");
     BF.duplicateConstantIslands();
-
-  const FunctionFragment FF = BF.getLayout().getFragment(
-      EmitColdPart ? FragmentNum::cold() : FragmentNum::hot());
+  }
 
   if (!FF.empty() && FF.front()->isLandingPad()) {
     assert(!FF.front()->isEntryPoint() &&
            "Landing pad cannot be entry point of function");
     // If the first block of the fragment is a landing pad, it's offset from the
     // start of the area that the corresponding LSDA describes is zero. In this
     // case, the call site entries in that LSDA have 0 as offset to the landing
     // pad, which the runtime interprets as "no handler". To prevent this,
for (auto I = BB->begin(), E = BB->end(); I != E; ++I) {
       }
 
       Streamer.emitInstruction(Instr, *BC.STI);
       LastIsPrefix = BC.MIB->isPrefix(Instr);
     }
   }
 
   if (!EmitCodeOnly)
-    emitConstantIslands(BF, EmitColdPart);
+    emitConstantIslands(BF, FF.isSplitFragment());
 }
 
 void BinaryEmitter::emitConstantIslands(BinaryFunction &BF, bool EmitColdPart,
                                         BinaryFunction *OnBehalfOf) {
   if (!BF.hasIslandsInfo())
     return;
 
   BinaryFunction::IslandInfo &Islands = BF.getIslandInfo();
void BinaryEmitter::emitLSDA(BinaryFunction &BF, bool EmitColdPart) {
   // Emit the LSDA label.
   MCSymbol *LSDASymbol =
       EmitColdPart ? BF.getColdLSDASymbol() : BF.getLSDASymbol();
   assert(LSDASymbol && "no LSDA symbol set");
   Streamer.emitLabel(LSDASymbol);
 
   // Corresponding FDE start.
   const MCSymbol *StartSymbol =
-      EmitColdPart ? BF.getColdSymbol() : BF.getSymbol();
+      EmitColdPart ? BF.getSymbol(FragmentNum::cold()) : BF.getSymbol();
 
   // Emit the LSDA header.
 
   // If LPStart is omitted, then the start of the FDE is used as a base for
   // landing pad displacements. Then if a cold fragment starts with
   // a landing pad, this means that the first landing pad offset will be 0.
   // As a result, the exception handling runtime will ignore this landing pad
   // because zero offset denotes the absence of a landing pad.

 namespace bolt {
 
 void emitBinaryContext(MCStreamer &Streamer, BinaryContext &BC,
                        StringRef OrgSecPrefix) {
   BinaryEmitter(Streamer, BC).emitAll(OrgSecPrefix);
 }
 
 void emitFunctionBody(MCStreamer &Streamer, BinaryFunction &BF,
-                      bool EmitColdPart, bool EmitCodeOnly) {
+                      const FunctionFragment &FF, bool EmitCodeOnly) {
   BinaryEmitter(Streamer, BF.getBinaryContext())
-      .emitFunctionBody(BF, EmitColdPart, EmitCodeOnly);
+      .emitFunctionBody(BF, FF, EmitCodeOnly);
 }
 
 } // namespace bolt
 } // namespace llvm

bolt/lib/Core/BinaryFunction.cpp

if (BB->succ_size() == 1) {
       if (TSuccessor == FSuccessor)
         BB->removeDuplicateConditionalSuccessor(CondBranch);
       if (!NextBB ||
           ((NextBB != TSuccessor || !IsSupported) && NextBB != FSuccessor)) {
         // If one of the branches is guaranteed to be "long" while the other
         // could be "short", then prioritize short for "taken". This will
         // generate a sequence 1 byte shorter on x86.
         if (IsSupported && BC.isX86() &&
-            TSuccessor->isCold() != FSuccessor->isCold() &&
-            BB->isCold() != TSuccessor->isCold()) {
+            TSuccessor->getFragmentNum() != FSuccessor->getFragmentNum() &&
+            BB->getFragmentNum() != TSuccessor->getFragmentNum()) {
           std::swap(TSuccessor, FSuccessor);
           {
             auto L = BC.scopeLock();
             MIB->reverseBranchCondition(*CondBranch, TSuccessor->getLabel(),
                                         Ctx);
           }
           BB->swapConditionalSuccessors();
         }
void BinaryFunction::updateOutputValues(const MCAsmLayout &Layout) {
   if (!isEmitted()) {
     assert(!isInjected() && "injected function should be emitted");
     setOutputAddress(getAddress());
     setOutputSize(getSize());
     return;
   }
 
   const uint64_t BaseAddress = getCodeSection()->getOutputAddress();
-  ErrorOr<BinarySection &> ColdSection = getColdCodeSection();
+  ErrorOr<BinarySection &> ColdSection =
+      getColdCodeSection(FragmentNum::cold());
   const uint64_t ColdBaseAddress =
       isSplit() ? ColdSection->getOutputAddress() : 0;
   if (BC.HasRelocations || isInjected()) {
     const uint64_t StartOffset = Layout.getSymbolOffset(*getSymbol());
     const uint64_t EndOffset = Layout.getSymbolOffset(*getFunctionEndLabel());
     setOutputAddress(BaseAddress + StartOffset);
     setOutputSize(EndOffset - StartOffset);
     if (hasConstantIsland()) {
       const uint64_t DataOffset =
           Layout.getSymbolOffset(*getFunctionConstantIslandLabel());
       setOutputDataAddress(BaseAddress + DataOffset);
     }
     if (isSplit()) {
-      const MCSymbol *ColdStartSymbol = getColdSymbol();
+      const MCSymbol *ColdStartSymbol = getSymbol(FragmentNum::cold());
       assert(ColdStartSymbol && ColdStartSymbol->isDefined() &&
              "split function should have defined cold symbol");
       const MCSymbol *ColdEndSymbol = getFunctionColdEndLabel();
       assert(ColdEndSymbol && ColdEndSymbol->isDefined() &&
              "split function should have defined cold end symbol");
       const uint64_t ColdStartOffset = Layout.getSymbolOffset(*ColdStartSymbol);
       const uint64_t ColdEndOffset = Layout.getSymbolOffset(*ColdEndSymbol);
       cold().setAddress(ColdBaseAddress + ColdStartOffset);

bolt/lib/Rewrite/RewriteInstance.cpp

 //===- bolt/Rewrite/RewriteInstance.cpp - ELF rewriter --------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "bolt/Rewrite/RewriteInstance.h"
 #include "bolt/Core/BinaryContext.h"
 #include "bolt/Core/BinaryEmitter.h"
 #include "bolt/Core/BinaryFunction.h"
 #include "bolt/Core/DebugData.h"
 #include "bolt/Core/Exceptions.h"
+#include "bolt/Core/FunctionLayout.h"
 #include "bolt/Core/MCPlusBuilder.h"
 #include "bolt/Core/ParallelUtilities.h"
 #include "bolt/Core/Relocation.h"
 #include "bolt/Passes/CacheMetrics.h"
 #include "bolt/Passes/ReorderFunctions.h"
 #include "bolt/Profile/BoltAddressTranslation.h"
 #include "bolt/Profile/DataAggregator.h"
 #include "bolt/Profile/DataReader.h"
if (Section &&
         (Function->getImageAddress() == 0 || Function->getImageSize() == 0))
       continue;
 
     // Restore origin section for functions that were emitted or supposed to
     // be emitted to patch sections.
     if (Section)
       BC->deregisterSection(*Section);
     assert(Function->getOriginSectionName() && "expected origin section");
-    Function->CodeSectionName = std::string(*Function->getOriginSectionName());
-    if (Function->isSplit()) {
-      if (ErrorOr<BinarySection &> ColdSection = Function->getColdCodeSection())
+    Function->CodeSectionName = Function->getOriginSectionName()->str();
+    for (const FunctionFragment FF :
+         Function->getLayout().getSplitFragments()) {
+      if (ErrorOr<BinarySection &> ColdSection =
+              Function->getColdCodeSection(FF.getFragmentNum()))
         BC->deregisterSection(*ColdSection);
-      Function->ColdCodeSectionName = std::string(getBOLTTextSectionName());
     }
+    if (Function->getLayout().isSplit())
+      Function->ColdCodeSectionName = getBOLTTextSectionName().str();
   }
 
   if (opts::PrintCacheMetrics) {
     outs() << "BOLT-INFO: cache metrics after emitting functions:\n";
     CacheMetrics::printAll(BC->getSortedFunctions());
   }
 
   if (opts::KeepTmp) {
if (opts::JumpTables == JTS_BASIC) {
                           << '\n');
         RTDyld.reassignSectionAddress(Section.getSectionID(), JT->getAddress());
       }
     }
 
     if (!Function.isSplit())
       continue;
 
-    ErrorOr<BinarySection &> ColdSection = Function.getColdCodeSection();
+    for (const FunctionFragment FF : Function.getLayout().getSplitFragments()) {
+      ErrorOr<BinarySection &> ColdSection =
+          Function.getColdCodeSection(FF.getFragmentNum());
       assert(ColdSection && "cannot find section for cold part");
       // Cold fragments are aligned at 16 bytes.
       NextAvailableAddress = alignTo(NextAvailableAddress, 16);
       BinaryFunction::FragmentInfo &ColdPart = Function.cold();
       if (TooLarge) {
         // The corresponding FDE will refer to address 0.
         ColdPart.setAddress(0);
         ColdPart.setImageAddress(0);
         ColdPart.setImageSize(0);
         ColdPart.setFileOffset(0);
       } else {
         ColdPart.setAddress(NextAvailableAddress);
         ColdPart.setImageAddress(ColdSection->getAllocAddress());
         ColdPart.setImageSize(ColdSection->getOutputSize());
         ColdPart.setFileOffset(getFileOffsetForAddress(NextAvailableAddress));
         ColdSection->setOutputAddress(ColdPart.getAddress());
       }
 
       LLVM_DEBUG(dbgs() << "BOLT: mapping cold fragment 0x"
                         << Twine::utohexstr(ColdPart.getImageAddress())
                         << " to 0x" << Twine::utohexstr(ColdPart.getAddress())
                         << " with size "
                         << Twine::utohexstr(ColdPart.getImageSize()) << '\n');
       RTDyld.reassignSectionAddress(ColdSection->getSectionID(),
                                     ColdPart.getAddress());
 
       NextAvailableAddress += ColdPart.getImageSize();
     }
+  }
 
   // Add the new text section aggregating all existing code sections.
   // This is pseudo-section that serves a purpose of creating a corresponding
   // entry in section header table.
   int64_t NewTextSectionSize =
       NextAvailableAddress - NewTextSectionStartAddress;
   if (NewTextSectionSize) {
     const unsigned Flags = BinarySection::getFlags(/*IsReadOnly=*/true,
if (Function.isFolded()) {
                           .concat(".icf.0")
                           .toStringRef(Buf));
       ICFSymbol.st_value = ICFParent->getOutputAddress();
       ICFSymbol.st_size = ICFParent->getOutputSize();
       ICFSymbol.st_shndx = ICFParent->getCodeSection()->getIndex();
       Symbols.emplace_back(ICFSymbol);
     }
     if (Function.isSplit() && Function.cold().getAddress()) {
+      for (const FunctionFragment FF :
+           Function.getLayout().getSplitFragments()) {
         ELFSymTy NewColdSym = FunctionSymbol;
-      SmallVector<char, 256> Buf;
-      NewColdSym.st_name =
-          AddToStrTab(Twine(cantFail(FunctionSymbol.getName(StringSection)))
-                          .concat(".cold.0")
-                          .toStringRef(Buf));
-      NewColdSym.st_shndx = Function.getColdCodeSection()->getIndex();
+        const SmallString<256> Buf = formatv(
+            "{0}.cold.{1}", cantFail(FunctionSymbol.getName(StringSection)),
+            FF.getFragmentNum().get() - 1);
+        NewColdSym.st_name = AddToStrTab(Buf);
+        NewColdSym.st_shndx =
+            Function.getColdCodeSection(FF.getFragmentNum())->getIndex();
         NewColdSym.st_value = Function.cold().getAddress();
         NewColdSym.st_size = Function.cold().getImageSize();
         NewColdSym.setBindingAndType(ELF::STB_LOCAL, ELF::STT_FUNC);
         Symbols.emplace_back(NewColdSym);
       }
+    }
     if (Function.hasConstantIsland()) {
       uint64_t DataMark = Function.getOutputDataAddress();
       uint64_t CISize = getConstantIslandSize(Function);
       uint64_t CodeMark = DataMark + CISize;
       ELFSymTy DataMarkSym = FunctionSymbol;
       DataMarkSym.st_name = AddToStrTab("$d");
       DataMarkSym.st_value = DataMark;
       DataMarkSym.st_size = 0;
if (Function) {
       Function =
           (Symbol.getType() == ELF::STT_FUNC)
               ? BC->getBinaryFunctionContainingAddress(Symbol.st_value,
                                                        /*CheckPastEnd=*/false,
                                                        /*UseMaxSize=*/true)
               : nullptr;
 
       if (Function && Function->isEmitted()) {
+        assert(Function->getLayout().isHotColdSplit() &&
+               "Adding symbols based on cold fragment when there are more than "
+               "2 fragments");
         const uint64_t OutputAddress =
             Function->translateInputToOutputAddress(Symbol.st_value);
 
         NewSymbol.st_value = OutputAddress;
         // Force secondary entry points to have zero size.
         NewSymbol.st_size = 0;
         NewSymbol.st_shndx =
             OutputAddress >= Function->cold().getAddress() &&
                     OutputAddress < Function->cold().getImageSize()
-                ? Function->getColdCodeSection()->getIndex()
+                ? Function->getColdCodeSection(FragmentNum::cold())->getIndex()
                 : Function->getCodeSection()->getIndex();
       } else {
         // Check if the symbol belongs to moved data object and update it.
         BinaryData *BD = opts::ReorderData.empty()
                              ? nullptr
                              : BC->getBinaryDataAtAddress(Symbol.st_value);
         if (BD && BD->isMoved() && !BD->isJumpTable()) {
           assert((!BD->getSize() || !Symbol.st_size ||