Propeller: LLVM support for basic block sections (Base Patch - Part 2)

[tmsriram]

Summary

This is part 2 of the base patch for LLVM Support of basic block sections. Part 1 is here: https://reviews.llvm.org/D68063

• This patch supports creating sections for all basic blocks or a subset of basic blocks. The subset of basic blocks via a clang option -fbasicblock-sections=<list_file> which contains a list of functions and the basic block ids for which sections are desired.

Please see https://github.com/google/llvm-propeller/blob/plo-dev/Propeller_RFC.pdf for a detailed discussion on basic block sections.

Labeling Basic Blocks :

Every basic block is labelled with a unique symbol as this allows easy mapping of virtual addresses from PMU profiles back to the corresponding basic blocks. Since the number of basic blocks is large, the labeling bloats the symbol table sizes and the string table sizes significantly. While the binary size does increase this does not affect performance as the symbol table is not loaded in memory during run-timea, the string table size bloat is kept very minimal using a unary naming scheme that uses string suffix compression. The basic blocks for function foo are named "a.bb.foo", "aa.bb.foo", . . . This turns out to be very good for string table sizes and the bloat in the string table size for a very large binary is only 8 %.

[tmsriram]

Add a test to make sure basic block sections are created.

[davidxl]

Can you add more test cases to cover things like bb labels, and different bb section types (cold, EH, unique etc). For cold section types, the merging should kick in etc.

llvm/lib/CodeGen/MachineFunction.cpp

371

Fix comment. Missing the verb

421

It is not clear from the code below that the same BB begins and ends it for unique sections.

[tmsriram]

Rename "BasicBlockSections" to "BBSections".

[tmsriram]

llvm/lib/CodeGen/MachineFunction.cpp

421

For unique sections, this comes already set above in line 388 and 389.

[davidxl]

llvm/lib/CodeGen/MachineFunction.cpp

421

ok. Perhaps add a comment or ASSERT here.

llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp

774

Add a comment on '0' parameter : 0 /*blah ..*/,

786

Is the logic reversed? Can you add a comment in the source?

807

Why there is no getUniqueSectioNames check here as in other cases?

[MaskRay]

(Not a reviewer, so just my armchair critic: this patch touches a lot of code but the test is tiny. I am concerned that many code paths may be untested.)

[tmsriram]

New tests for basicblock-sections=labels and basicblock-sections=<file>

[tmsriram]

One more test to check if sections can be created only for a subset of basic blocks within the function.

[tmsriram]

One more test to make sure exception sections are created as expected.

[tmsriram]

Code refactor and address comments.

llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp

786

It is not reversed. With unique section names, the section for function main would be named .text.main.cold and will be unique across this module as main is a unique name. Without unique section names, this would be named just .text.cold and all cold sections in this module would be grouped in one section.

So, the function name must be appended to section only if -funique-section-names is off. I have refactored this code and combined two functions into one.

807

Good catch, fixed.

[davidxl]

Also missing a test case on cold basic blocks.

llvm/lib/CodeGen/MachineBasicBlock.cpp

74

Is this the best encoding scheme? I suppose in many cases, a single byte is enough? (e.g. total number of blocks in a func is small). Or this allows better string compression?

llvm/test/CodeGen/X86/basicblock-sections-eh.ll

84 ↗

(On Diff #242133)

Make test more strict to test landing pad blocks are in the same section?

llvm/test/CodeGen/X86/basicblock-sections-listbb.ll

35 ↗

(On Diff #242133)

how to tell if this is the id 2 section?

[tmsriram]

llvm/lib/CodeGen/MachineBasicBlock.cpp

74

This works great for ".strtab" size and we tried a few other simple encoding schemes like "1.BB", "2.BB", etc. While the simple encoding schemes are small, what we have is much smaller. The ".strtab" bloat even when turned on for all basic blocks is only 8%. The symtab bloat remains the same irrespective of the encoding. Please note that symtab and strtab are not ALLOCed and do not affect performance.

llvm/test/CodeGen/X86/basicblock-sections-eh.ll

84 ↗

(On Diff #242133)

The character "l" denotes if this is a landing pad or not. Here there is only one. I can modify it to have more landing pads.

llvm/test/CodeGen/X86/basicblock-sections-listbb.ll

35 ↗

(On Diff #242133)

The id is the count of the number of characters before ".BB". For instance, this is "r.BB._Z3bazb" and is the "id 1" basic block. That's why we explicitly check in the line before that this does not get a section. The entry block is "id 0'.

[tmsriram]

Add another test to check for cold sections.

[efriedma]

I'm afraid the way this patch is written, we're going to have to substantially rewrite it to port to other targets. In particular, we're assigning blocks to sections in the asmprinter, which is way too late for targets that do branch relaxation as an MIR pass.

Is there some reason to emit "cold" and "exception" blocks into separate sections? Not sure what benefit you get from separating them.

llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3030

I'm confused what MF->front().isExceptionSection() is supposed to check. The entry block can't be an exception handler, I think?

3042

I don't think the linker treats the ".cold" suffix specially. Maybe we should do something different? (There's special support in binutils ld for the prefix ".text.unlikely."; not sure how useful that would be in this context.)

llvm/lib/CodeGen/MachineFunction.cpp

362

Can this actually get called more than once?

Why do we care if the blocks are sorted? Does it help optimize the code somehow?

[rahmanl]

llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3030

Great catch. This block should be removed. We'll remove it in the next update.

[tmsriram]

In D73674#1868439, @efriedma wrote:

I'm afraid the way this patch is written, we're going to have to substantially rewrite it to port to other targets. In particular, we're assigning blocks to sections in the asmprinter, which is way too late for targets that do branch relaxation as an MIR pass.

Hi Eli, What do you suggest we do here? An MIR pass that assigns sections to basic blocks?

Is there some reason to emit "cold" and "exception" blocks into separate sections? Not sure what benefit you get from separating them.

Sections containing hot blocks can be grouped together, function splitting of cold basic blocks and placing all the cold blocks together and away from the hot blocks gives us better iTLB utilization.

[efriedma]

In D73674#1877267, @tmsriram wrote:

In D73674#1868439, @efriedma wrote:

I'm afraid the way this patch is written, we're going to have to substantially rewrite it to port to other targets. In particular, we're assigning blocks to sections in the asmprinter, which is way too late for targets that do branch relaxation as an MIR pass.

Hi Eli, What do you suggest we do here? An MIR pass that assigns sections to basic blocks?

Yes, that would make sense. Not completely sure where it should run, but we can adjust that later. Maybe we want it to happen before MachineBlockPlacement, so we can do section-aware block placement?

Is there some reason to emit "cold" and "exception" blocks into separate sections? Not sure what benefit you get from separating them.

Sections containing hot blocks can be grouped together, function splitting of cold basic blocks and placing all the cold blocks together and away from the hot blocks gives us better iTLB utilization.

I meant, is there some reason to emit "cold" and "exception" sections separately from each other. I understand the benefit of separating both of them from the hot blocks.

[tmsriram]

In D73674#1877341, @efriedma wrote:

In D73674#1877267, @tmsriram wrote:

In D73674#1868439, @efriedma wrote:

I'm afraid the way this patch is written, we're going to have to substantially rewrite it to port to other targets. In particular, we're assigning blocks to sections in the asmprinter, which is way too late for targets that do branch relaxation as an MIR pass.

Hi Eli, What do you suggest we do here? An MIR pass that assigns sections to basic blocks?

Yes, that would make sense. Not completely sure where it should run, but we can adjust that later. Maybe we want it to happen before MachineBlockPlacement, so we can do section-aware block placement?

Is there some reason to emit "cold" and "exception" blocks into separate sections? Not sure what benefit you get from separating them.

Sections containing hot blocks can be grouped together, function splitting of cold basic blocks and placing all the cold blocks together and away from the hot blocks gives us better iTLB utilization.

I meant, is there some reason to emit "cold" and "exception" sections separately from each other. I understand the benefit of separating both of them from the hot blocks.

Apologies, I misread the original question. Yes, in cases where exception blocks are hot, it does not help placing them along with cold blocks. One of the SPEC benchmarks had a performance improvement doing this.

[efriedma]

In D73674#1877420, @tmsriram wrote:

Apologies, I misread the original question. Yes, in cases where exception blocks are hot, it does not help placing them along with cold blocks. One of the SPEC benchmarks had a performance improvement doing this.

Oh, okay. And I guess you can't emit multiple exception sections due to some arcane exception-handling stuff? (I remember you mentioned this at some point, but I don't remember the details.) Therefore the compromise is emitting them separately from the cold blocks? I guess that's reasonable.

[rahmanl]

In D73674#1868439, @efriedma wrote:

Is there some reason to emit "cold" and "exception" blocks into separate sections? Not sure what benefit you get from separating them.

Great point Eli!
As @tmsriram mentioned, in the case where hot eh pads are present, this allows us to reorder the exception section and place it in the hot part.
However, I modified the code such that if all eh pads are cold, we don't create a separate exception section and group all eh pads together with the cold section. The next patch update by @tmsriram will include this change. A test will also be included in D73739.

[tmsriram]

Create a new pass and assign sections to Machine Basic Blocks in this pass.

Address reviewer comments, one more test.

[tmsriram]

In D73674#1877420, @tmsriram wrote:

In D73674#1877341, @efriedma wrote:

In D73674#1877267, @tmsriram wrote:

In D73674#1868439, @efriedma wrote:

I'm afraid the way this patch is written, we're going to have to substantially rewrite it to port to other targets. In particular, we're assigning blocks to sections in the asmprinter, which is way too late for targets that do branch relaxation as an MIR pass.

Hi Eli, What do you suggest we do here? An MIR pass that assigns sections to basic blocks?

Yes, that would make sense. Not completely sure where it should run, but we can adjust that later. Maybe we want it to happen before MachineBlockPlacement, so we can do section-aware block placement?

Created a new pass, BBSectionsPrepare.cpp that does all the work of assigning sections to MBBs. This removes all code from CodeGenPrepare.cpp and Function.h.

Is there some reason to emit "cold" and "exception" blocks into separate sections? Not sure what benefit you get from separating them.

Sections containing hot blocks can be grouped together, function splitting of cold basic blocks and placing all the cold blocks together and away from the hot blocks gives us better iTLB utilization.

I meant, is there some reason to emit "cold" and "exception" sections separately from each other. I understand the benefit of separating both of them from the hot blocks.

Apologies, I misread the original question. Yes, in cases where exception blocks are hot, it does not help placing them along with cold blocks. One of the SPEC benchmarks had a performance improvement doing this.

[tmsriram]

llvm/lib/CodeGen/MachineFunction.cpp

362

Having basic blocks in the same section together makes it really easy to update CFI and DebugInfo (like label differences). It also makes the .s more readable.

[efriedma]

llvm/include/llvm/CodeGen/MachineBasicBlock.h

133

I'd like to have MIR support for serializing/deserializing the SectionType, so we can test the BBSectionsPrepare pass separately from the passes that run after it and the asmprinter.

[tmsriram]

llvm/include/llvm/CodeGen/MachineBasicBlock.h

133

• Function attribute is one way I suppose but if the sections need to be created only for a subset of basic blocks then we would need to emit metadata?

• I thought of emitting a function attribute like "bbsections" with a string value which encodes the basic block ids that require sections.

Thoughts?

[efriedma]

llvm/include/llvm/CodeGen/MachineBasicBlock.h

133

We can print/parse data for MachineBasicBlocks directly. We do this for other bits of data stored on MachineBasicBlock. See MIPrinter::print(const MachineBasicBlock &MBB).

[tmsriram]

Add support for serializing Basic Block Section Type.

• Print machine basic block section type with MIR Printer
• Parse Section Type from MIR
• Add tests for -stop-after=bbsections-prepare and -start-after=bbsections-prepare to test bbsections-prepare pass.

[efriedma]

LGTM with one minor comment.

llvm/lib/CodeGen/MIRParser/MIParser.cpp

638 ↗

(On Diff #247545)

error()? This code is pretty clearly reachable.

[tmsriram]

Rebase, address reviewer comments.

• Change llvm_unreachable to error in MIParser.cpp, make getMBBS return bool.

[tmsriram]

Rebase

• Change to use MemoryBuffer rather than string for bbsections file
• Detailed comments in BBSectionsPrepare.cpp on basic block sections

[thakis]

This breaks tests on non-Linux.

mac: http://45.33.8.238/mac/10215/step_11.txt
win: http://45.33.8.238/win/10569/step_11.txt

llvm/include/llvm/CodeGen/MachineBasicBlock.h

private:
   bool IsEHScopeEntry = false;
 
   /// Indicate that this basic block is the entry block of an EH funclet.
   bool IsEHFuncletEntry = false;
 
   /// Indicate that this basic block is the entry block of a cleanup funclet.
   bool IsCleanupFuncletEntry = false;
 
+  /// Indicate that this basic block begins a new section.
+  bool IsBeginSection = false;
+
+  /// Indicate that this basic block ends the current section.
+  bool IsEndSection = false;
+
+  /// Indicate that this basic block belongs to the cold section.
+  bool IsColdSection = false;
+
+  /// Indicate that this basic block belong to the exception section.
+  bool IsExceptionSection = false;
+
   /// since getSymbol is a relatively heavy-weight operation, the symbol
   /// is only computed once and is cached.
   mutable MCSymbol *CachedMCSymbol = nullptr;
 
+  /// Used during basic block sections to mark the end of a basic block.
+  MCSymbol *EndMCSymbol = nullptr;
+
   // Intrusive list support
   MachineBasicBlock() = default;
 
   explicit MachineBasicBlock(MachineFunction &MF, const BasicBlock *BB);
 
   ~MachineBasicBlock();
 
   // MachineBasicBlocks are allocated and owned by MachineFunction.
#endif
   void setIsEHFuncletEntry(bool V = true) { IsEHFuncletEntry = V; }
 
   /// Returns true if this is the entry block of a cleanup funclet.
   bool isCleanupFuncletEntry() const { return IsCleanupFuncletEntry; }
 
   /// Indicates if this is the entry block of a cleanup funclet.
   void setIsCleanupFuncletEntry(bool V = true) { IsCleanupFuncletEntry = V; }
 
+  /// Returns true if this block begins a section.
+  bool isBeginSection() const { return IsBeginSection; }
+
+  /// Indicate that this basic block begins a new section.
+  void setBeginSection(bool V = true) { IsBeginSection = V; }
+
+  /// Returns true if this block ends a section.
+  bool isEndSection() const { return IsEndSection; }
+
+  /// Indicate that this basic block ends a section.
+  void setEndSection(bool V = true) { IsEndSection = V; }
+
+  /// Returns true if this basic block belongs to the cold section.
+  bool isColdSection() const { return IsColdSection; }
+
+  /// Indicate that this basic block belongs to the cold section.
+  void setColdSection(bool V = true) { IsColdSection = V; }
+
+  /// Returns true if this basic block belongs to the exception section.
+  bool isExceptionSection() const { return IsExceptionSection; }
+
+  /// Indicate that this basic block belongs to the exception section.
+  void setExceptionSection(bool V = true) { IsExceptionSection = V; }
+
   /// Returns true if it is legal to hoist instructions into this block.
   bool isLegalToHoistInto() const;
 
   // Code Layout methods.
 
   /// Move 'this' block before or after the specified block.  This only moves
   /// the block, it does not modify the CFG or adjust potential fall-throughs at
   /// the end of the block.
   void moveBefore(MachineBasicBlock *NewAfter);
   void moveAfter(MachineBasicBlock *NewBefore);
 
+  /// Insert an unconditional jump to a fallthrough block if any.
+  void insertUnconditionalFallthroughBranch();
+
+  /// Returns true if this and MBB belong to the same section.
+  bool sameSection(const MachineBasicBlock *MBB) const;
+
+  /// Returns the basic block that ends the section which contains this one.
+  const MachineBasicBlock *getSectionEndMBB() const;
+
   /// Update the terminator instructions in block to account for changes to the
   /// layout. If the block previously used a fallthrough, it may now need a
   /// branch, and if it previously used branching it may now be able to use a
   /// fallthrough.
   void updateTerminator();
 
   // Machine-CFG mutators
 
#endif
   /// MachineBasicBlocks are uniquely numbered at the function level, unless
   /// they're not in a MachineFunction yet, in which case this will return -1.
   int getNumber() const { return Number; }
   void setNumber(int N) { Number = N; }
 
   /// Return the MCSymbol for this basic block.
   MCSymbol *getSymbol() const;
 
+  /// Sets the MCSymbol corresponding to the end of this basic block.
+  void setEndMCSymbol(MCSymbol *Sym) { EndMCSymbol = Sym; }
+
+  /// Returns the MCSymbol corresponding to the end of this basic block.
+  MCSymbol *getEndMCSymbol() const { return EndMCSymbol; }
+
   Optional<uint64_t> getIrrLoopHeaderWeight() const {
     return IrrLoopHeaderWeight;
   }
 
   void setIrrLoopHeaderWeight(uint64_t Weight) {
     IrrLoopHeaderWeight = Weight;
   }
 

llvm/include/llvm/CodeGen/MachineFunction.h

 class MachineFrameInfo;
 class MachineFunction;
 class MachineJumpTableInfo;
 class MachineModuleInfo;
 class MachineRegisterInfo;
 class MCContext;
 class MCInstrDesc;
 class MCSymbol;
+class MCSection;
 class Pass;
 class PseudoSourceValueManager;
 class raw_ostream;
 class SlotIndexes;
 class TargetRegisterClass;
 class TargetSubtargetInfo;
 struct WasmEHFuncInfo;
 struct WinEHFuncInfo;
class MachineFunction {
   MachineFrameInfo *FrameInfo;
 
   // Keep track of constants which are spilled to memory
   MachineConstantPool *ConstantPool;
 
   // Keep track of jump tables for switch instructions
   MachineJumpTableInfo *JumpTableInfo;
 
+  // Keep track of the function section.
+  MCSection *Section = nullptr;
+
   // Keeps track of Wasm exception handling related data. This will be null for
   // functions that aren't using a wasm EH personality.
   WasmEHFuncInfo *WasmEHInfo = nullptr;
 
   // Keeps track of Windows exception handling related data. This will be null
   // for functions that aren't using a funclet-based EH personality.
   WinEHFuncInfo *WinEHInfo = nullptr;
 
   // Function-level unique numbering for MachineBasicBlocks.  When a
   // MachineBasicBlock is inserted into a MachineFunction is it automatically
   // numbered and this vector keeps track of the mapping from ID's to MBB's.
   std::vector<MachineBasicBlock*> MBBNumbering;
 
+  // Unary encoding of basic block symbols is used to reduce size of ".strtab".
+  // Basic block number 'i' gets a prefix of length 'i'.  The ith character also
+  // denotes the type of basic block number 'i'.  Return blocks are marked with
+  // 'r', landing pads with 'l' and regular blocks with 'a'.
+  std::vector<char> MBBSymbolPrefix;
+
   // Pool-allocate MachineFunction-lifetime and IR objects.
   BumpPtrAllocator Allocator;
 
   // Allocation management for instructions in function.
   Recycler<MachineInstr> InstructionRecycler;
 
   // Allocation management for operand arrays on instructions.
   ArrayRecycler<MachineOperand> OperandRecycler;
class MachineFunction {
   /// CodeView label annotations.
   std::vector<std::pair<MCSymbol *, MDNode *>> CodeViewAnnotations;
 
   bool CallsEHReturn = false;
   bool CallsUnwindInit = false;
   bool HasEHScopes = false;
   bool HasEHFunclets = false;
 
+  // True if basic block sections that are generated have been sorted.
+  bool BBSectionsSorted = false;
+  // True if sections must be generated for all basic blocks.
+  bool BasicBlockSections = false;
+  // True if labels must be generated for all basic blocks.
+  bool BasicBlockLabels = false;
+
   /// List of C++ TypeInfo used.
   std::vector<const GlobalValue *> TypeInfos;
 
   /// List of typeids encoding filters used.
   std::vector<unsigned> FilterIds;
 
   /// List of the indices in FilterIds corresponding to filter terminators.
   std::vector<unsigned> FilterEnds;
assert(delegate && !TheDelegate &&
            "first resetting it!");
 
     TheDelegate = delegate;
   }
 
   MachineModuleInfo &getMMI() const { return MMI; }
   MCContext &getContext() const { return Ctx; }
 
+  /// Returns the Section this function belongs to.
+  MCSection *getSection() const { return Section; }
+
+  /// Indicates the Section this function belongs to.
+  void setSection(MCSection *S) { Section = S; }
+
   PseudoSourceValueManager &getPSVManager() const { return *PSVManager; }
 
   /// Return the DataLayout attached to the Module associated to this MF.
   const DataLayout &getDataLayout() const;
 
   /// Return the LLVM function that this machine code represents
   const Function &getFunction() const { return F; }
 
   /// getName - Return the name of the corresponding LLVM function.
   StringRef getName() const;
 
   /// getFunctionNumber - Return a unique ID for the current function.
   unsigned getFunctionNumber() const { return FunctionNumber; }
 
+  /// Returns true if this function has basic block sections enabled.
+  bool getBasicBlockSections() const { return BasicBlockSections; }
+
+  /// Sort the basic blocks according to the sections they belong to.
+  bool sortBasicBlockSections();
+
+  /// Indicates that basic block Labels are to be generated for this function.
+  void setBasicBlockLabels();
+
+  /// Returns true if basic block labels are to be generated for this function.
+  bool getBasicBlockLabels() const { return BasicBlockLabels; }
+
   /// getTarget - Return the target machine this machine code is compiled with
   const LLVMTargetMachine &getTarget() const { return Target; }
 
   /// getSubtarget - Return the subtarget for which this machine code is being
   /// compiled.
   const TargetSubtargetInfo &getSubtarget() const { return *STI; }
 
   /// getSubtarget - This method returns a pointer to the specified type of
public:
   /// instruction from the instruction stream.
   void eraseCallSiteInfo(const MachineInstr *MI);
 
   /// Copy the call site info from \p Old to \ New. Its usage is when we are
   /// making a copy of the instruction that will be inserted at different point
   /// of the instruction stream.
   void copyCallSiteInfo(const MachineInstr *Old,
                         const MachineInstr *New);
+
+  const std::vector<char> &getMBBSymbolPrefix() const {
+    return MBBSymbolPrefix;
+  }
 };
 
 //===--------------------------------------------------------------------===//
 // GraphTraits specializations for function basic block graphs (CFGs)
 //===--------------------------------------------------------------------===//
 
 // Provide specializations of GraphTraits to be able to treat a
 // machine function as a graph of machine basic blocks... these are

llvm/include/llvm/CodeGen/TargetLoweringObjectFileImpl.h

MCSection *getExplicitSectionGlobal(const GlobalObject *GO, SectionKind Kind,
                                       const TargetMachine &TM) const override;
 
   MCSection *SelectSectionForGlobal(const GlobalObject *GO, SectionKind Kind,
                                     const TargetMachine &TM) const override;
 
   MCSection *getSectionForJumpTable(const Function &F,
                                     const TargetMachine &TM) const override;
 
+  MCSection *
+  getSectionForMachineBasicBlock(const Function &F,
+                                 const MachineBasicBlock &MBB,
+                                 const TargetMachine &TM) const override;
+
+  MCSection *
+  getColdSectionForMachineBasicBlock(const Function &F,
+                                     const MachineBasicBlock &MBB,
+                                     const TargetMachine &TM) const override;
+
+  MCSection *
+  getEHSectionForMachineBasicBlock(const Function &F,
+                                   const MachineBasicBlock &MBB,
+                                   const TargetMachine &TM) const override;
+
   bool shouldPutJumpTableInFunctionSection(bool UsesLabelDifference,
                                            const Function &F) const override;
 
   /// Return an MCExpr to use for a reference to the specified type info global
   /// variable from exception handling information.
   const MCExpr *getTTypeGlobalReference(const GlobalValue *GV,
                                         unsigned Encoding,
                                         const TargetMachine &TM,

llvm/include/llvm/Target/TargetLoweringObjectFile.h

 #include "llvm/IR/Module.h"
 #include "llvm/MC/MCObjectFileInfo.h"
 #include "llvm/MC/SectionKind.h"
 #include <cstdint>
 
 namespace llvm {
 
 class GlobalValue;
+class MachineBasicBlock;
 class MachineModuleInfo;
 class Mangler;
 class MCContext;
 class MCExpr;
 class MCSection;
 class MCSymbol;
 class MCSymbolRefExpr;
 class MCStreamer;
public:
 
   /// Given a constant with the SectionKind, return a section that it should be
   /// placed in.
   virtual MCSection *getSectionForConstant(const DataLayout &DL,
                                            SectionKind Kind,
                                            const Constant *C,
                                            unsigned &Align) const;
 
+  virtual MCSection *
+  getSectionForMachineBasicBlock(const Function &F,
+                                 const MachineBasicBlock &MBB,
+                                 const TargetMachine &TM) const;
+
+  virtual MCSection *
+  getColdSectionForMachineBasicBlock(const Function &F,
+                                     const MachineBasicBlock &MBB,
+                                     const TargetMachine &TM) const;
+
+  virtual MCSection *
+  getEHSectionForMachineBasicBlock(const Function &F,
+                                   const MachineBasicBlock &MBB,
+                                   const TargetMachine &TM) const;
+
   /// Classify the specified global variable into a set of target independent
   /// categories embodied in SectionKind.
   static SectionKind getKindForGlobal(const GlobalObject *GO,
                                       const TargetMachine &TM);
 
   /// This method computes the appropriate section to emit the specified global
   /// variable or function definition. This should not be passed external (or
   /// available externally) globals.

llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp

if (isVerbose())
     OutStreamer->GetCommentOS()
         << "-- Begin function "
         << GlobalValue::dropLLVMManglingEscape(F.getName()) << '\n';
 
   // Print out constants referenced by the function
   EmitConstantPool();
 
   // Print the 'header' of function.
-  OutStreamer->SwitchSection(getObjFileLowering().SectionForGlobal(&F, TM));
+  MF->setSection(getObjFileLowering().SectionForGlobal(&F, TM));
+  OutStreamer->SwitchSection(MF->getSection());
+
   EmitVisibility(CurrentFnSym, F.getVisibility());
 
   if (MAI->needsFunctionDescriptors() &&
       F.getLinkage() != GlobalValue::InternalLinkage)
     EmitLinkage(&F, CurrentFnDescSym);
 
   EmitLinkage(&F, CurrentFnSym);
   if (MAI->hasFunctionAlignment())
if (!MLI) {
       OwnedMLI->getBase().analyze(MDT->getBase());
       MLI = OwnedMLI.get();
     }
   }
 
   // Print out code for the function.
   bool HasAnyRealCode = false;
   int NumInstsInFunction = 0;
+  bool emitBasicBlockSections = MF->getBasicBlockSections();
+  MachineBasicBlock *EndOfRegularSectionMBB = nullptr;
+  bool emitBBLabels = MF->getBasicBlockSections() || MF->getBasicBlockLabels();
+  if (emitBBLabels)
+    MF->setBasicBlockLabels();
+  if (emitBasicBlockSections) {
+    MF->sortBasicBlockSections();
+    EndOfRegularSectionMBB =
+        const_cast<MachineBasicBlock *>(MF->front().getSectionEndMBB());
+    assert(EndOfRegularSectionMBB->isEndSection() &&
+           "The MBB at the end of the regular section must end a section");
+  }
+
   for (auto &MBB : *MF) {
     // Print a label for the basic block.
     EmitBasicBlockStart(MBB);
     for (auto &MI : MBB) {
       // Print the assembly for the instruction.
       if (!MI.isPosition() && !MI.isImplicitDef() && !MI.isKill() &&
           !MI.isDebugInstr()) {
         HasAnyRealCode = true;
for (auto &MI : MBB) {
         for (const HandlerInfo &HI : Handlers) {
           NamedRegionTimer T(HI.TimerName, HI.TimerDescription,
                              HI.TimerGroupName, HI.TimerGroupDescription,
                              TimePassesIsEnabled);
           HI.Handler->endInstruction();
         }
       }
     }
-
+    if (&MBB != EndOfRegularSectionMBB &&
+        (MF->getBasicBlockLabels() || MBB.isEndSection())) {
+      // Emit size directive for the size of this basic block.  Create a symbol
+      // for the end of the basic block.
+      MCSymbol *CurrentBBEnd = OutContext.createTempSymbol();
+      const MCExpr *SizeExp = MCBinaryExpr::createSub(
+          MCSymbolRefExpr::create(CurrentBBEnd, OutContext),
+          MCSymbolRefExpr::create(MBB.getSymbol(), OutContext), OutContext);
+      OutStreamer->EmitLabel(CurrentBBEnd);
+      MBB.setEndMCSymbol(CurrentBBEnd);
+      OutStreamer->emitELFSize(MBB.getSymbol(), SizeExp);
+    }
     EmitBasicBlockEnd(MBB);
   }
 
   EmittedInsts += NumInstsInFunction;
   MachineOptimizationRemarkAnalysis R(DEBUG_TYPE, "InstructionCount",
                                       MF->getFunction().getSubprogram(),
                                       &MF->front());
   R << ore::NV("NumInstructions", NumInstsInFunction)
if (!HasAnyRealCode && (MAI->hasSubsectionsViaSymbols() ||
     // Targets can opt-out of emitting the noop here by leaving the opcode
     // unspecified.
     if (Noop.getOpcode()) {
       OutStreamer->AddComment("avoids zero-length function");
       OutStreamer->EmitInstruction(Noop, getSubtargetInfo());
     }
   }
 
+  // Switch to the original section if basic block sections was used.
+  if (emitBasicBlockSections)
+    OutStreamer->SwitchSection(MF->getSection());
+
   const Function &F = MF->getFunction();
   for (const auto &BB : F) {
     if (!BB.hasAddressTaken())
       continue;
     MCSymbol *Sym = GetBlockAddressSymbol(&BB);
     if (Sym->isDefined())
       continue;
     OutStreamer->AddComment("Address of block that was removed by CodeGen");
     OutStreamer->EmitLabel(Sym);
   }
 
   // Emit target-specific gunk after the function body.
   EmitFunctionBodyEnd();
 
   if (needFuncLabelsForEHOrDebugInfo(*MF, MMI) ||
-      MAI->hasDotTypeDotSizeDirective()) {
+      MAI->hasDotTypeDotSizeDirective() || emitBasicBlockSections) {
     // Create a symbol for the end of function.
     CurrentFnEnd = createTempSymbol("func_end");
     OutStreamer->EmitLabel(CurrentFnEnd);
   }
 
   // If the target wants a .size directive for the size of the function, emit
   // it.
   if (MAI->hasDotTypeDotSizeDirective()) {
     // We can get the size as difference between the function label and the
     // temp label.
     const MCExpr *SizeExp = MCBinaryExpr::createSub(
         MCSymbolRefExpr::create(CurrentFnEnd, OutContext),
         MCSymbolRefExpr::create(CurrentFnSymForSize, OutContext), OutContext);
     OutStreamer->emitELFSize(CurrentFnSym, SizeExp);
   }
 
   for (const HandlerInfo &HI : Handlers) {
     NamedRegionTimer T(HI.TimerName, HI.TimerDescription, HI.TimerGroupName,
                        HI.TimerGroupDescription, TimePassesIsEnabled);
     HI.Handler->markFunctionEnd();
   }
 
+  if (emitBasicBlockSections)
+    EndOfRegularSectionMBB->setEndMCSymbol(CurrentFnEnd);
+
   // Print out jump tables referenced by the function.
   EmitJumpTableInfo();
 
   // Emit post-function debug and/or EH information.
   for (const HandlerInfo &HI : Handlers) {
     NamedRegionTimer T(HI.TimerName, HI.TimerDescription, HI.TimerGroupName,
                        HI.TimerGroupDescription, TimePassesIsEnabled);
     HI.Handler->endFunction(MF);
static void emitBasicBlockLoopComments(const MachineBasicBlock &MBB,
 
   PrintChildLoopComment(OS, Loop, AP.getFunctionNumber());
 }
 
 /// EmitBasicBlockStart - This method prints the label for the specified
 /// MachineBasicBlock, an alignment (if present) and a comment describing
 /// it if appropriate.
 void AsmPrinter::EmitBasicBlockStart(const MachineBasicBlock &MBB) {
+  bool BasicBlockSections = MF->getBasicBlockSections();
   // End the previous funclet and start a new one.
   if (MBB.isEHFuncletEntry()) {
     for (const HandlerInfo &HI : Handlers) {
       HI.Handler->endFunclet();
       HI.Handler->beginFunclet(MBB);
     }
   }
 
   // Emit an alignment directive for this block, if needed.
+  if (MBB.pred_empty() || !BasicBlockSections) {
     const Align Alignment = MBB.getAlignment();
     if (Alignment != Align::None())
       EmitAlignment(Alignment);
+  }
 
   // If the block has its address taken, emit any labels that were used to
   // reference the block.  It is possible that there is more than one label
   // here, because multiple LLVM BB's may have been RAUW'd to this block after
   // the references were generated.
   if (MBB.hasAddressTaken()) {
     const BasicBlock *BB = MBB.getBasicBlock();
     if (isVerbose())
if (const BasicBlock *BB = MBB.getBasicBlock()) {
         OutStreamer->GetCommentOS() << '\n';
       }
     }
 
     assert(MLI != nullptr && "MachineLoopInfo should has been computed");
     emitBasicBlockLoopComments(MBB, MLI, *this);
   }
 
-  // Print the main label for the block.
+  bool emitBBLabels = BasicBlockSections || MF->getBasicBlockLabels();
   if (MBB.pred_empty() ||
-      (isBlockOnlyReachableByFallthrough(&MBB) && !MBB.isEHFuncletEntry() &&
-       !MBB.hasLabelMustBeEmitted())) {
+      (!emitBBLabels && isBlockOnlyReachableByFallthrough(&MBB) &&
+       !MBB.isEHFuncletEntry() && !MBB.hasLabelMustBeEmitted())) {
     if (isVerbose()) {
       // NOTE: Want this comment at start of line, don't emit with AddComment.
       OutStreamer->emitRawComment(" %bb." + Twine(MBB.getNumber()) + ":",
                                   false);
     }
   } else {
-    if (isVerbose() && MBB.hasLabelMustBeEmitted())
+    if (isVerbose() && MBB.hasLabelMustBeEmitted()) {
       OutStreamer->AddComment("Label of block must be emitted");
+    }
+    // With -fbasicblock-sections, a basic block can start a new section.
+    if (MBB.isExceptionSection()) {
+      if (MF->front().isExceptionSection()) {
+        OutStreamer->SwitchSection(MF->getSection());
+      } else {
+        OutStreamer->SwitchSection(
+            getObjFileLowering().getEHSectionForMachineBasicBlock(
+                MF->getFunction(), MBB, TM));
+      }
+    } else if (MBB.isColdSection()) {
+      // Create the cold section here.
+      OutStreamer->SwitchSection(
+          getObjFileLowering().getColdSectionForMachineBasicBlock(
+              MF->getFunction(), MBB, TM));
+    } else if (MBB.isBeginSection() && MBB.isEndSection()) {
+      OutStreamer->SwitchSection(
+          getObjFileLowering().getSectionForMachineBasicBlock(MF->getFunction(),
+                                                              MBB, TM));
+    } else if (BasicBlockSections) {
+      OutStreamer->SwitchSection(MF->getSection());
+    }
     OutStreamer->EmitLabel(MBB.getSymbol());
   }
 }
 
 
 void AsmPrinter::EmitVisibility(MCSymbol *Sym, unsigned Visibility,
                                 bool IsDefinition) const {
   MCSymbolAttr Attr = MCSA_Invalid;
if (Attr != MCSA_Invalid)
     OutStreamer->EmitSymbolAttribute(Sym, Attr);
 }
 
 /// isBlockOnlyReachableByFallthough - Return true if the basic block has
 /// exactly one predecessor and the control transfer mechanism between
 /// the predecessor and this block is a fall-through.
 bool AsmPrinter::
 isBlockOnlyReachableByFallthrough(const MachineBasicBlock *MBB) const {
+  // With BasicBlock Sections, no block is a fall through.
+  if (MBB->isBeginSection())
+    return false;
+
   // If this is a landing pad, it isn't a fall through.  If it has no preds,
   // then nothing falls through to it.
   if (MBB->isEHPad() || MBB->pred_empty())
     return false;
 
   // If there isn't exactly one predecessor, it can't be a fall through.
   if (MBB->pred_size() > 1)
     return false;

llvm/lib/CodeGen/MachineBasicBlock.cpp

 #include "llvm/IR/ModuleSlotTracker.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCContext.h"
 #include "llvm/Support/DataTypes.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Target/TargetMachine.h"
 #include <algorithm>
+#include <cstdlib>
 using namespace llvm;
 
 #define DEBUG_TYPE "codegen"
 
 static cl::opt<bool> PrintSlotIndexes(
     "print-slotindexes",
     cl::desc("When printing machine IR, annotate instructions and blocks with "
              "SlotIndexes when available"),

 }
 
 /// Return the MCSymbol for this basic block.
 MCSymbol *MachineBasicBlock::getSymbol() const {
   if (!CachedMCSymbol) {
     const MachineFunction *MF = getParent();
     MCContext &Ctx = MF->getContext();
     auto Prefix = Ctx.getAsmInfo()->getPrivateLabelPrefix();
+
+    bool BasicBlockSymbols =
+        MF->getBasicBlockSections() || MF->getBasicBlockLabels();
+    auto Delimiter = BasicBlockSymbols ? "." : "_";
     assert(getNumber() >= 0 && "cannot get label for unreachable MBB");
-    CachedMCSymbol = Ctx.getOrCreateSymbol(Twine(Prefix) + "BB" +
-                                           Twine(MF->getFunctionNumber()) +
-                                           "_" + Twine(getNumber()));
-  }
 
+    // With Basic Block Sections, we emit a symbol for every basic block. To
+    // keep the size of strtab small, we choose a unary encoding which can
+    // compress the symbol names significantly.  The basic blocks for function
+    // foo are named a.BB.foo, aa.BB.foo, and so on.
+    if (BasicBlockSymbols) {
+      auto Iter = MF->getMBBSymbolPrefix().begin();
+      if (getNumber() < 0 ||
+          getNumber() >= (int)MF->getMBBSymbolPrefix().size())
+        report_fatal_error("Unreachable MBB: " + Twine(getNumber()));
+      std::string Prefix(Iter + 1, Iter + getNumber() + 1);
+      std::reverse(Prefix.begin(), Prefix.end());
+      CachedMCSymbol =
+          Ctx.getOrCreateSymbol(Prefix + Twine(Delimiter) + "BB" +
+                                Twine(Delimiter) + Twine(MF->getName()));
+    } else {
+      CachedMCSymbol = Ctx.getOrCreateSymbol(
+          Twine(Prefix) + "BB" + Twine(MF->getFunctionNumber()) +
+          Twine(Delimiter) + Twine(getNumber()));
+    }
+  }
   return CachedMCSymbol;
 }
 
 
 raw_ostream &llvm::operator<<(raw_ostream &OS, const MachineBasicBlock &MBB) {
   MBB.print(OS);
   return OS;
 }

 void MachineBasicBlock::moveBefore(MachineBasicBlock *NewAfter) {
   getParent()->splice(NewAfter->getIterator(), getIterator());
 }
 
 void MachineBasicBlock::moveAfter(MachineBasicBlock *NewBefore) {
   getParent()->splice(++NewBefore->getIterator(), getIterator());
 }
 
+// Returns true if this basic block and the Other are in the same section.
+bool MachineBasicBlock::sameSection(const MachineBasicBlock *Other) const {
+  if (this == Other)
+    return true;
+
+  if ((this->isColdSection() != Other->isColdSection()) ||
+      (this->isExceptionSection() != Other->isExceptionSection()))
+    return false;
+
+  if ((this->isBeginSection() && this->isEndSection()) ||
+      (Other->isBeginSection() && Other->isEndSection()))
+    return false;
+
+  return true;
+}
+
+const MachineBasicBlock *MachineBasicBlock::getSectionEndMBB() const {
+  if (this->isEndSection())
+    return this;
+  auto I = std::next(this->getIterator());
+  const MachineFunction *MF = getParent();
+  while (I != MF->end()) {
+    const MachineBasicBlock &MBB = *I;
+    if (MBB.isEndSection())
+      return &MBB;
+    I = std::next(I);
+  }
+  llvm_unreachable("No End Basic Block for this section.");
+}
+
+// Insert unconditional jumps to the basic block to which there is
+// a fall through.
+void MachineBasicBlock::insertUnconditionalFallthroughBranch() {
+  MachineBasicBlock *Fallthrough = getFallThrough();
+
+  if (Fallthrough == nullptr)
+    return;
+
+  // If this basic block and the Fallthrough basic block are in the same
+  // section then do not insert the jump.
+  if (this->sameSection(Fallthrough))
+    return;
+
+  const TargetInstrInfo *TII = getParent()->getSubtarget().getInstrInfo();
+  SmallVector<MachineOperand, 4> Cond;
+  MachineBasicBlock *TBB = nullptr, *FBB = nullptr;
+
+  // If a branch to the fall through block already exists, return.
+  if (!TII->analyzeBranch(*this, TBB, FBB, Cond) &&
+      (TBB == Fallthrough || FBB == Fallthrough)) {
+    return;
+  }
+
+  Cond.clear();
+  DebugLoc DL = findBranchDebugLoc();
+  TII->insertBranch(*this, Fallthrough, nullptr, Cond, DL);
+}
+
 void MachineBasicBlock::updateTerminator() {
   const TargetInstrInfo *TII = getParent()->getSubtarget().getInstrInfo();
   // A block with no successors has no concerns with fall-through edges.
   if (this->succ_empty())
     return;
 
   MachineBasicBlock *TBB = nullptr, *FBB = nullptr;
   SmallVector<MachineOperand, 4> Cond;

llvm/lib/CodeGen/MachineFunction.cpp

 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineJumpTableInfo.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/PseudoSourceValue.h"
 #include "llvm/CodeGen/TargetFrameLowering.h"
+#include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetLowering.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/CodeGen/WasmEHFuncInfo.h"
 #include "llvm/CodeGen/WinEHFuncInfo.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/BasicBlock.h"
void MachineFunction::init() {
   Alignment = STI->getTargetLowering()->getMinFunctionAlignment();
 
   // FIXME: Shouldn't use pref alignment if explicit alignment is set on F.
   // FIXME: Use Function::hasOptSize().
   if (!F.hasFnAttribute(Attribute::OptimizeForSize))
     Alignment = std::max(Alignment,
                          STI->getTargetLowering()->getPrefFunctionAlignment());
 
+  // Check if basic block sections are required for this function.
+  if (Target.getBasicBlockSections() == llvm::BasicBlockSection::All ||
+      F.getBasicBlockSections() ||
+      (Target.getBasicBlockSections() == llvm::BasicBlockSection::List &&
+       Target.isFunctionInBasicBlockSectionsList(F.getName())))
+    BasicBlockSections = true;
+
+  // Check if basic block labels are required for this function.
+  if (Target.getBasicBlockSections() == llvm::BasicBlockSection::Labels ||
+      F.getBasicBlockLabels())
+    BasicBlockLabels = true;
+
   if (AlignAllFunctions)
     Alignment = Align(1ULL << AlignAllFunctions);
 
   JumpTableInfo = nullptr;
 
   if (isFuncletEHPersonality(classifyEHPersonality(
           F.hasPersonalityFn() ? F.getPersonalityFn() : nullptr))) {
     WinEHInfo = new (Allocator) WinEHFuncInfo();
void MachineFunction::RenumberBlocks(MachineBasicBlock *MBB) {
   }
 
   // Okay, all the blocks are renumbered.  If we have compactified the block
   // numbering, shrink MBBNumbering now.
   assert(BlockNo <= MBBNumbering.size() && "Mismatch!");
   MBBNumbering.resize(BlockNo);
 }
 
+/// This function sorts basic blocks according to the sections in which they are
+/// emitted.  Basic block sections automatically turn on function sections so
+/// the entry block is in the function section.  The other sections that are
+/// created are:
+/// 1) Exception section - basic blocks that are landing pads
+/// 2) Cold section - basic blocks that will not have unique sections.
+/// 3) Unique section - one per basic block that is emitted in a unique section.
+bool MachineFunction::sortBasicBlockSections() {
+  // This should only be done once no matter how many times it is called.
+  if (this->BBSectionsSorted || !this->getBasicBlockSections())
+    return false;
+
+  DenseMap<const MachineBasicBlock *, unsigned> MBBOrder;
+  unsigned MBBOrderN = 0;
+
+  SmallSet<unsigned, 4> S = Target.getBasicBlockSectionsSet(F.getName());
+  for (auto &MBB : *this) {
+    // A unique BB section can only be created if this basic block is not
+    // used for exception table computations.  Entry basic block cannot
+    // a section because the function starts one.
+    if (MBB.getNumber() == this->front().getNumber()) {
+      if (MBB.isEHPad())
+        MBB.setExceptionSection();
+      continue;
+    }
+    // Also, check if this BB is a cold basic block in which case sections
+    // are not required with the list option.
+    bool isColdBB =
+        ((Target.getBasicBlockSections() == llvm::BasicBlockSection::List) &&
+         !S.count(MBB.getNumber()));
+    if (MBB.isEHPad()) {
+      MBB.setExceptionSection();
+    } else if (isColdBB) {
+      MBB.setColdSection();
+    } else {
+      MBB.setBeginSection();
+      MBB.setEndSection();
+    }
+    MBBOrder[&MBB] = MBBOrderN++;
+  }
+
+  // With -fbasicblock-sections, fall through blocks must be made
+  // explicitly reachable.  Do this after sections is set as
+  // unnecessary fallthroughs can be avoided.
+  for (auto &MBB : *this) {
+    MBB.insertUnconditionalFallthroughBranch();
+  }
+
+  // Order : Entry Block, Exception Section, Cold Section,
+  // Other Unique Sections.
+  auto SectionType = ([&](MachineBasicBlock &X) {
+    if (X.getNumber() == this->front().getNumber() && !X.isExceptionSection())
+      return 0;
+    if (X.isExceptionSection())
+      return 1;
+    else if (X.isColdSection())
+      return 2;
+    return 3;
+  });
+
+  this->sort(([&](MachineBasicBlock &X, MachineBasicBlock &Y) {
+    auto TypeX = SectionType(X);
+    auto TypeY = SectionType(Y);
+
+    return (TypeX != TypeY) ? TypeX < TypeY : MBBOrder[&X] < MBBOrder[&Y];
+  }));
+
+  // Set the basic block that begins or ends every section.  For unique
+  // sections, the same basic block begins and ends it.
+  MachineBasicBlock *PrevMBB = nullptr;
+  for (auto &MBB : *this) {
+    // Entry block
+    if (MBB.getNumber() == this->front().getNumber()) {
+      PrevMBB = &MBB;
+      continue;
+    }
+    assert(PrevMBB != nullptr && "First block was not a regular block!");
+    int TypeP = SectionType(*PrevMBB);
+    int TypeT = SectionType(MBB);
+    if (TypeP != TypeT) {
+      PrevMBB->setEndSection();
+      MBB.setBeginSection();
+    }
+    PrevMBB = &MBB;
+  }
+  PrevMBB->setEndSection();
+
+  this->BBSectionsSorted = true;
+  return true;
+}
+
+/// This is used with -fbasicblock-sections or -fbasicblock-labels option.
+/// A unary encoding of basic block labels is done to keep ".strtab" sizes
+/// small.
+void MachineFunction::setBasicBlockLabels() {
+  const TargetInstrInfo *TII = getSubtarget().getInstrInfo();
+  this->MBBSymbolPrefix.resize(getNumBlockIDs(), 'a');
+  for (auto MBBI = begin(), E = end(); MBBI != E; ++MBBI) {
+    if (MBBI->getNumber() < 0 || MBBI->getNumber() >= (int)getNumBlockIDs())
+      report_fatal_error(
+          "BasicBlock number was out of range: " + Twine(MBBI->getNumber()) +
+          " [0 -> " + Twine(getNumBlockIDs()) + "]");
+    // 'a' - Normal block.
+    // 'r' - Return block.
+    // 'l' - Landing Pad.
+    // 'L' - Return and landing pad.
+    bool isEHPad = MBBI->isEHPad();
+    bool isRetBlock = MBBI->isReturnBlock() && !TII->isTailCall(MBBI->back());
+    char type = 'a';
+    if (isEHPad && isRetBlock)
+      type = 'L';
+    else if (isEHPad)
+      type = 'l';
+    else if (isRetBlock)
+      type = 'r';
+    MBBSymbolPrefix[MBBI->getNumber()] = type;
+  }
+}
+
 /// Allocate a new MachineInstr. Use this instead of `new MachineInstr'.
 MachineInstr *MachineFunction::CreateMachineInstr(const MCInstrDesc &MCID,
                                                   const DebugLoc &DL,
                                                   bool NoImp) {
   return new (InstructionRecycler.Allocate<MachineInstr>(Allocator))
     MachineInstr(*this, MCID, DL, NoImp);
 }
 

llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp

 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/BinaryFormat/COFF.h"
 #include "llvm/BinaryFormat/Dwarf.h"
 #include "llvm/BinaryFormat/ELF.h"
 #include "llvm/BinaryFormat/MachO.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachineModuleInfoImpls.h"
 #include "llvm/IR/Comdat.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalAlias.h"

 #include "llvm/MC/MCStreamer.h"
 #include "llvm/MC/MCSymbol.h"
 #include "llvm/MC/MCSymbolELF.h"
 #include "llvm/MC/MCValue.h"
 #include "llvm/MC/SectionKind.h"
 #include "llvm/ProfileData/InstrProf.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CodeGen.h"
-#include "llvm/Support/Format.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/Format.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Target/TargetMachine.h"
 #include <cassert>
 #include <string>
 
 using namespace llvm;
 using namespace dwarf;
 
if (Kind.isMergeableConst32() && MergeableConst32Section)
     return MergeableConst32Section;
   if (Kind.isReadOnly())
     return ReadOnlySection;
 
   assert(Kind.isReadOnlyWithRel() && "Unknown section kind");
   return DataRelROSection;
 }
 
+/// Returns a unique section for the given machine basic block.
+MCSection *TargetLoweringObjectFileELF::getSectionForMachineBasicBlock(
+    const Function &F, const MachineBasicBlock &MBB,
+    const TargetMachine &TM) const {
+  SmallString<128> Name;
+  Name = (static_cast<MCSectionELF *>(MBB.getParent()->getSection()))
+             ->getSectionName();
+  if (TM.getUniqueBBSectionNames()) {
+    Name += ".";
+    Name += MBB.getSymbol()->getName();
+  }
+  unsigned UniqueID = NextUniqueID++;
+  unsigned Flags = ELF::SHF_ALLOC | ELF::SHF_EXECINSTR;
+  std::string GroupName = "";
+  if (F.hasComdat()) {
+    Flags |= ELF::SHF_GROUP;
+    GroupName = F.getComdat()->getName();
+  }
+  return getContext().getELFSection(Name, ELF::SHT_PROGBITS, Flags, 0,
+                                    GroupName, UniqueID);
+}
+
+/// Returns the cold section in which this basic block belongs.
+MCSection *TargetLoweringObjectFileELF::getColdSectionForMachineBasicBlock(
+    const Function &F, const MachineBasicBlock &MBB,
+    const TargetMachine &TM) const {
+  SmallString<128> Name;
+  Name = (static_cast<MCSectionELF *>(MBB.getParent()->getSection()))
+             ->getSectionName();
+
+  if (!TM.getUniqueSectionNames()) {
+    Name += ".";
+    Name += MBB.getParent()->getName();
+  }
+
+  Name += ".cold";
+
+  unsigned Flags = ELF::SHF_ALLOC | ELF::SHF_EXECINSTR;
+  std::string GroupName = "";
+  if (F.hasComdat()) {
+    Flags |= ELF::SHF_GROUP;
+    GroupName = F.getComdat()->getName();
+  }
+  return getContext().getELFSection(Name, ELF::SHT_PROGBITS, Flags, 0,
+                                    GroupName);
+}
+
+/// Returns the exception section in which this basic block belongs.
+MCSection *TargetLoweringObjectFileELF::getEHSectionForMachineBasicBlock(
+    const Function &F, const MachineBasicBlock &MBB,
+    const TargetMachine &TM) const {
+  SmallString<128> Name;
+  Name = (static_cast<MCSectionELF *>(MBB.getParent()->getSection()))
+             ->getSectionName();
+  Name += ".eh";
+
+  unsigned Flags = ELF::SHF_ALLOC | ELF::SHF_EXECINSTR;
+  std::string GroupName = "";
+  if (F.hasComdat()) {
+    Flags |= ELF::SHF_GROUP;
+    GroupName = F.getComdat()->getName();
+  }
+  return getContext().getELFSection(Name, ELF::SHT_PROGBITS, Flags, 0,
+                                    GroupName);
+}
+
 static MCSectionELF *getStaticStructorSection(MCContext &Ctx, bool UseInitArray,
                                               bool IsCtor, unsigned Priority,
                                               const MCSymbol *KeySym) {
   std::string Name;
   unsigned Type;
   unsigned Flags = ELF::SHF_ALLOC | ELF::SHF_WRITE;
   StringRef COMDAT = KeySym ? KeySym->getName() : "";
 

llvm/lib/Target/TargetLoweringObjectFile.cpp

SectionKind TargetLoweringObjectFile::getKindForGlobal(const GlobalObject *GO,
                                                        const TargetMachine &TM){
   assert(!GO->isDeclaration() && !GO->hasAvailableExternallyLinkage() &&
          "Can only be used for global definitions");
 
   // Functions are classified as text sections.
   if (isa<Function>(GO))
     return SectionKind::getText();
 
+  // Basic blocks are classified as text sections.
+  if (isa<BasicBlock>(GO))
+    return SectionKind::getText();
+
   // Global variables require more detailed analysis.
   const auto *GVar = cast<GlobalVariable>(GO);
 
   // Handle thread-local data first.
   if (GVar->isThreadLocal()) {
     if (isSuitableForBSS(GVar) && !TM.Options.NoZerosInBSS)
       return SectionKind::getThreadBSS();
     return SectionKind::getThreadData();
MCSection *TargetLoweringObjectFile::getSectionForConstant(
     const DataLayout &DL, SectionKind Kind, const Constant *C,
     unsigned &Align) const {
   if (Kind.isReadOnly() && ReadOnlySection != nullptr)
     return ReadOnlySection;
 
   return DataSection;
 }
 
+MCSection *TargetLoweringObjectFile::getSectionForMachineBasicBlock(
+    const Function &F, const MachineBasicBlock &MBB,
+    const TargetMachine &TM) const {
+  return nullptr;
+}
+
+MCSection *TargetLoweringObjectFile::getColdSectionForMachineBasicBlock(
+    const Function &F, const MachineBasicBlock &MBB,
+    const TargetMachine &TM) const {
+  return nullptr;
+}
+
+MCSection *TargetLoweringObjectFile::getEHSectionForMachineBasicBlock(
+    const Function &F, const MachineBasicBlock &MBB,
+    const TargetMachine &TM) const {
+  return nullptr;
+}
 /// getTTypeGlobalReference - Return an MCExpr to use for a
 /// reference to the specified global variable from exception
 /// handling information.
 const MCExpr *TargetLoweringObjectFile::getTTypeGlobalReference(
     const GlobalValue *GV, unsigned Encoding, const TargetMachine &TM,
     MachineModuleInfo *MMI, MCStreamer &Streamer) const {
   const MCSymbolRefExpr *Ref =
       MCSymbolRefExpr::create(TM.getSymbol(GV), getContext());

llvm/test/CodeGen/X86/basicblock-sections.ll

+; RUN: llc < %s -mtriple=x86_64-pc-linux -function-sections -basicblock-sections=all -unique-bb-section-names | FileCheck %s -check-prefix=LINUX-SECTIONS
+; RUN: llc < %s -mtriple=i386-unknown-linux-gnu  -function-sections -basicblock-sections=all -unique-bb-section-names | FileCheck %s -check-prefix=LINUX-SECTIONS
+
+define void @_Z3bazb(i1 zeroext) {
+  %2 = alloca i8, align 1
+  %3 = zext i1 %0 to i8
+  store i8 %3, i8* %2, align 1
+  %4 = load i8, i8* %2, align 1
+  %5 = trunc i8 %4 to i1
+  br i1 %5, label %6, label %8
+
+6:                                                ; preds = %1
+  %7 = call i32 @_Z3barv()
+  br label %10
+
+8:                                                ; preds = %1
+  %9 = call i32 @_Z3foov()
+  br label %10
+
+10:                                               ; preds = %8, %6
+  ret void
+}
+
+declare i32 @_Z3barv() #1
+
+declare i32 @_Z3foov() #1
+
+
+; LINUX-SECTIONS: .section        .text._Z3bazb,"ax",@progbits
+; LINUX-SECTIONS: _Z3bazb:
+; LINUX-SECTIONS: .section        .text._Z3bazb.r.BB._Z3bazb,"ax",@progbits,unique,1
+; LINUX-SECTIONS: r.BB._Z3bazb:
+; LINUX-SECTIONS: .size   r.BB._Z3bazb, .Ltmp0-r.BB._Z3bazb
+; LINUX-SECTIONS: .section        .text._Z3bazb.rr.BB._Z3bazb,"ax",@progbits,unique,2
+; LINUX-SECTIONS: rr.BB._Z3bazb:
+; LINUX-SECTIONS: .size   rr.BB._Z3bazb, .Ltmp1-rr.BB._Z3bazb