[BOLT] RFC: Add golang support

[yota9]

Summary

This patch adds golang binaries support built with original golang
compiler (gc). The golang support requers 3 separate optimization
passes:

• golang-preprocess: This pass must be executed before any changes were

perfomed in BinaryFunctions. Its main purpose is to parse the initial
metadata for golang functions.

• golang-postprocess: This pass must be executed after all optimizations

were performed (except for the LongJmp) pass. Its main purpose is too
adjust the code to sync it with functions metadata.

• golang: This pass must be executed after all code changes were

performed, no code changes are allowed by this pass or after it. This
pass only changes golang-specific data sections that contains functions
metadata.

Golang compiler patches:
Aarch64 support needs mapping symbols patch support, the PR was created
here: https://go-review.googlesource.com/c/go/+/343150 . The patches for
the gc supported versions might be found here:
https://github.com/yota9/golang_aarch64_mapping_symbols . Apply the
patch for your gc version and rebuilt the compiler.

Golang binaries compilation:
The golang default linker does not support emit-relocs option, the
external linker must be used for that purpose. The compilation command
examples:

• x86_64 (pie): go build -a -buildmode=pie -ldflags='-linkmode=external -extld=gcc -extldflags "-fuse-ld=gold -fPIE -pie -Wl,--threads -Wl,--emit-relocs -Wl,--compress-debug-sections=none"
• x86_64 (no-pie): go build -a -buildmode=exe -ldflags='-linkmode=external -extld=gcc -extldflags "-fuse-ld=gold -no-pie -Wl,--threads -Wl,--emit-relocs -Wl,--compress-debug-sections=none"
• aarch64 (non-pie): go build -a -ldflags='-linkmode=external -extld=gcc -extldflags "-no-pie -Wl,--emit-relocs -Wl,--compress-debug-sections=none"

NOTE: The aarch64 ld and gold linkers currently has some bugs related to PIE binaries. The ld emits wrong addresses in data sections, which does not affect runtime due to dynamic relocations, but affects our binary data extractions. The gold linker emits static relocations on the wrong offsets.

Restrictions:

• The golang support is tightly connected to the gc version, so we only

can optimize supported by BOLT gc versions compiled binaries.

• The inlining, frame optimization, split functions, lite mode

optimizations are unsupported.

Results:
Up to 19% of performance improvements were received on internal services.

NOTE: Golang binaries are compatible with instrumentation!

Testing:
TBD, probably a few binaries would be add in bolt-tests repo for both
x86 and aarch64 platforms.

Golang fixes related to support:
https://go-review.googlesource.com/c/go/+/396797
https://go-review.googlesource.com/c/go/+/380894
https://go-review.googlesource.com/c/go/+/334789

AArch64 gold linker veneer fix:
https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=cde010e1a866e67b7e895cbcb95dedd3de0a1e56;hp=a686428a8bb4cc6a32a976e36f1ec59aa5a9f58e

Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei

[yota9]

Some places are explained and marked by me that needs extra attention or some suggestions for re-implemetation

bolt/include/bolt/Core/BinarySection.h

447

The mechanism of in-place data section change should be re-implemented, your suggestions are welcomed :)

bolt/lib/Core/BinaryBasicBlock.cpp

638

We need this, since some of the instructions annotations might be "locked". E.g. when dealing with gc inlined functions, we might inline only one instruction (e.g NOP) which will be removed by bolt. But this instruction has inline metadata that we need to have in order to restore inlining golang tables. So before removing such instruction we need to create NOP instruction on it's place that will store this metadata.

bolt/lib/Core/BinaryEmitter.cpp

1169

The extra sections creation and emission mechanism should be re-implemented, your suggestions are welcomed :)

bolt/lib/Passes/Golang.cpp

1041

The mmap is used since at this point we are unable to tell the final section size, we are allocating 8 times more of original space, which is more than enough. Since mmap allocates pages lazily we don't have extra memory consumption. However we probalbe should use ordinary new here

bolt/lib/Passes/Instrumentation.cpp

188

The golang has a sp bias table for each function. Since instrumentation uses the stack we need to mark inserted instructions so later in golang-postprocess and golang pass we will handle these instructions specially

[yota9]

clang format warn

[tschuett]

This is a large patch.

Do you have plans to add Go support to llvm-objdump for testing?

[ayermolo]

Tests?
More specifically. Would it be possible to add unit tests, and not just binaries to bolt-tests?

[yota9]

@tschuett What do you mean by add go to llvm-objdump?
@ayermolo 'm not sure if I will be able to cover it compeletely or even all corner cases, probably only basic moments, but I will try to. As a first step I suggest to discuss technical point of the patch, there are some places that surely should be redone, I need some suggestions on this :) I don't think it will be fast process due to the patch size, my personal workload & etc,, but during that time I will try to add some test cases :)
UPD Could somebody please point me out what is the problem with git-clang-format stage? Locally It does not show any problem and I don't see any comments in the diff too, no idea what it is complaining about :(

[tschuett]

Using llvm-objdump to read properties of Go binaries.

[yota9]

@tschuett please checkout the answer above, I'm not sure if i will use llvm-objdump or will use other methods right now.

bolt/lib/Passes/BinaryPasses.cpp

1219

The problem here that if we set hot section based on the hasProfile insetad of hasValidIndex we will break LongJmp (tentativeLayoutRelocMode checks index), so we need to remove hasProfile case at all here (I will remove it with the next reupload). AFAIR the problem here is that if the function has profile but don't have exec count (which is not set if the main entry point does not have profile information for some reason) we would and up with split function with invalid index both parts of which will be emitted one by one in cold section. So we need to fix either function reordering alg to set hot index if the function has a profile (not exec count) or add check during profile read so that if the function hasPprofile() set exec count to be at least 1. Looking for your suggestions here too, probably I will separate this part from this commit to another one

[Amir]

Vladislav, thank you for adding this support! It's a massive change, so let's make it easier to review. Can you please split this diff up into small isolated chunks?
I would suggest having separate diffs for:

1. Core interface changes/additions
2. Instrumentation changes
3. MCPlusBuilder changes
4. Passes changes
5. Base Golang passes/functions/data structures
6. Interaction between Golang stuff and existing passes
7. Documentation changes

An example of such a stack would be D124900.

It would be much easier to integrate the work into the codebase with bite-sized diffs.

[yota9]

Rebase & some updates

[yota9]

Hello @Amir ! As you suggested I'm starting to splitting this patch to the parts. I suggest not to use stack of changes, it would be more convenient for me to post group of changes one-by-one, you can refer to the rest of the changes on this patch. As I understand your idea although smaller changes by it self are not very useful we still can commit them one by one due to the size of this patch. So the first change was prepared for core: https://reviews.llvm.org/D141234 . Thank you!

[yota9]

rebase, use new instead of mmap

[yota9]

Removs BS isChanged

[yota9]

fix tests

[yota9]

remove IsChanged

bolt/CODE_OWNERS.TXT

 D: All parts not covered by someone else
 
 N: Alexander Yermolovich
 E: [e-mail]
 D: DWARF support
 
 N: Vladislav Khmelevsky
 E: [e-mail]
-D: AArch64 backend
+D: AArch64 backend, Golang support

bolt/README.md

 Clang and GCC compilers. The main requirement for C/C++ code is not to rely
 on code layout properties, such as function pointer deltas.
 Assembly code can be processed too. Requirements for it include a clear
 separation of code and data, with data objects being placed into data
 sections/segments. If indirect jumps are used for intra-function control
 transfer (e.g., jump tables), the code patterns should be matching those
 generated by Clang/GCC.
 
+BOLT supports golang binaries built with the gc compiler using the
+`-golang=version` option, where the `version` is the go compiler version that
+was used to built the binary or `1` to automatically determine it based on
+the input file. Since golang support is strictly dependent on the compiler
+version BOLT won't be able to optimize currenly unsupported versions.
+
 NOTE: BOLT is currently incompatible with the `-freorder-blocks-and-partition`
 compiler option. Since GCC8 enables this option by default, you have to
 explicitly disable it by adding `-fno-reorder-blocks-and-partition` flag if
 you are compiling with GCC8 or above.
 
 NOTE2: DWARF v5 is the new debugging format generated by the latest LLVM and GCC
 compilers. It offers several benefits over the previous DWARF v4. Currently, the
 support for v5 is a work in progress for BOLT. While you will be able to

bolt/include/bolt/Core/BinaryBasicBlock.h

public:
 
   /// Erase pseudo instruction at a given iterator.
   /// Return iterator following the removed instruction.
   iterator erasePseudoInstruction(iterator II) {
     --NumPseudos;
     return Instructions.erase(II);
   }
 
+  /// Store meta-data of instruction to NOP instruction
+  /// and insert it in place of erased one
+  bool storeInstructionMetadata(iterator II);
+
   /// Erase non-pseudo instruction at a given iterator \p II.
   /// Return iterator following the removed instruction.
   iterator eraseInstruction(iterator II) {
+    if (storeInstructionMetadata(II))
+      return std::next(II);
+
     adjustNumPseudos(*II, -1);
     return Instructions.erase(II);
   }
 
   /// Erase non-pseudo instruction at a given \p Index
   void eraseInstructionAtIndex(unsigned Index) {
     eraseInstruction(Instructions.begin() + Index);
   }
public:
   }
 
   /// Replace instruction referenced by iterator \II with a sequence of
   /// instructions defined by [\p Begin, \p End] range.
   ///
   /// Return iterator pointing to the first inserted instruction.
   template <typename Itr>
   iterator replaceInstruction(iterator II, Itr Begin, Itr End) {
-    adjustNumPseudos(*II, -1);
+    II = eraseInstruction(II);
     adjustNumPseudos(Begin, End, 1);
-
-    auto I = II - Instructions.begin();
-    Instructions.insert(Instructions.erase(II), Begin, End);
-    return I + Instructions.begin();
+    return Instructions.insert(II, Begin, End);
   }
 
   iterator replaceInstruction(iterator II,
                               const InstructionListType &Replacement) {
     return replaceInstruction(II, Replacement.begin(), Replacement.end());
   }
 
   /// Insert \p NewInst before \p At, which must be an existing instruction in
   /// this BB. Return iterator pointing to the newly inserted instruction.
   iterator insertInstruction(iterator At, MCInst &&NewInst) {
     adjustNumPseudos(NewInst, 1);
     return Instructions.emplace(At, std::move(NewInst));
   }
 
   iterator insertInstruction(iterator At, MCInst &NewInst) {
     adjustNumPseudos(NewInst, 1);
     return Instructions.insert(At, NewInst);
   }
 
+  iterator insertInstructions(iterator At, std::vector<MCInst> &Instrs) {
+    for (MCInst &NewInst : Instrs) {
+      At = insertInstruction(At, NewInst);
+      ++At;
+    }
+    return At;
+  }
+
   /// Helper to retrieve any terminators in \p BB before \p Pos. This is used
   /// to skip CFI instructions and to retrieve the first terminator instruction
   /// in basic blocks with two terminators (conditional jump and unconditional
   /// jump).
   MCInst *getTerminatorBefore(MCInst *Pos);
 
   /// Used to identify whether an instruction is before a terminator and whether
   /// moving it to the end of the BB would render it dead code.

bolt/include/bolt/Core/BinaryContext.h

 #define BOLT_CORE_BINARY_CONTEXT_H
 
 #include "bolt/Core/BinaryData.h"
 #include "bolt/Core/BinarySection.h"
 #include "bolt/Core/DebugData.h"
 #include "bolt/Core/JumpTable.h"
 #include "bolt/Core/MCPlusBuilder.h"
 #include "bolt/RuntimeLibs/RuntimeLibrary.h"
+#include "bolt/Utils/NameResolver.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/iterator.h"
 #include "llvm/BinaryFormat/Dwarf.h"
 #include "llvm/BinaryFormat/MachO.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCCodeEmitter.h"
 #include "llvm/MC/MCContext.h"
std::map<uint64_t, BinaryFunction> &getBinaryFunctions() {
     return BinaryFunctions;
   }
 
   /// Return all functions for this rewrite instance.
   const std::map<uint64_t, BinaryFunction> &getBinaryFunctions() const {
     return BinaryFunctions;
   }
 
+  /// Return BF by name. If the global function was not found return the first
+  /// found local
+  BinaryFunction *getBinaryFunctionByName(StringRef Name) {
+    BinaryData *Data = getFirstBinaryDataByName(Name);
+    if (!Data)
+      return nullptr;
+
+    return getBinaryFunctionAtAddress(Data->getAddress());
+  }
+
+  const BinaryFunction *getBinaryFunctionByName(StringRef Name) const {
+    return const_cast<BinaryContext *>(this)->getBinaryFunctionByName(Name);
+  }
+
   /// Create BOLT-injected function
   BinaryFunction *createInjectedBinaryFunction(const std::string &Name,
                                                bool IsSimple = true);
 
   std::vector<BinaryFunction *> &getInjectedBinaryFunctions() {
     return InjectedBinaryFunctions;
   }
 
const BinaryData *getPLTBinaryDataByName(StringRef Name) const {
     // without it.
     size_t At = Name.find("@");
     if (At != std::string::npos)
       return getBinaryDataByName(Name.str().substr(0, At) + "@PLT");
 
     return nullptr;
   }
 
+  /// Return BinaryData for the given \p Name
+  /// If the data was not found return first local BinaryData or nullptr
+  BinaryData *getFirstBinaryDataByName(StringRef Name) {
+    BinaryData *Data = getBinaryDataByName(Name);
+    if (!Data)
+      return getBinaryDataByName(NameResolver::uniquifyID(Name, 1));
+    return Data;
+  }
+
+  const BinaryData *getFirstBinaryDataByName(StringRef Name) const {
+    return const_cast<BinaryContext *>(this)->getFirstBinaryDataByName(Name);
+  }
+
   /// Return true if \p SymbolName was generated internally and was not present
   /// in the input binary.
   bool isInternalSymbolName(const StringRef Name) {
     return Name.startswith("SYMBOLat") || Name.startswith("DATAat") ||
            Name.startswith("HOLEat");
   }
 
   MCSymbol *getHotTextStartSymbol() const {
std::pair<size_t, size_t> calculateEmittedSize(BinaryFunction &BF,
                                                  bool FixBranches = true);
 
   /// Calculate the size of the instruction \p Inst optionally using a
   /// user-supplied emitter for lock-free multi-thread work. MCCodeEmitter is
   /// not thread safe and each thread should operate with its own copy of it.
   uint64_t
   computeInstructionSize(const MCInst &Inst,
                          const MCCodeEmitter *Emitter = nullptr) const {
+    if (isAArch64())
+      return 4;
+
     if (auto Size = MIB->getAnnotationWithDefault<uint32_t>(Inst, "Size"))
       return Size;
 
     if (!Emitter)
       Emitter = this->MCE.get();
     SmallString<256> Code;
     SmallVector<MCFixup, 4> Fixups;
     raw_svector_ostream VecOS(Code);
createStreamer(llvm::raw_pwrite_stream &OS) const {
     std::unique_ptr<MCStreamer> Streamer(TheTarget->createMCObjectStreamer(
         *TheTriple, *Ctx, std::unique_ptr<MCAsmBackend>(MAB), std::move(OW),
         std::unique_ptr<MCCodeEmitter>(MCE), *STI,
         /* RelaxAll */ false,
         /* IncrementalLinkerCompatible */ false,
         /* DWARFMustBeAtTheEnd */ false));
     return Streamer;
   }
+
+  /// Returns std pair of binary functons desired and maximum alignment bytes
+  std::pair<unsigned, unsigned> getBFAlignment(BinaryFunction &Function,
+                                               bool EmitColdPart) const;
 };
 
 template <typename T, typename = std::enable_if_t<sizeof(T) == 1>>
 inline raw_ostream &operator<<(raw_ostream &OS, const ArrayRef<T> &ByteArray) {
   const char *Sep = "";
   for (const auto Byte : ByteArray) {
     OS << Sep << format("%.2x", Byte);
     Sep = " ";
   }
   return OS;
 }
 
 } // namespace bolt
 } // namespace llvm
 
 #endif

bolt/include/bolt/Core/BinaryData.h

protected:
 
   /// Section this data belongs to.
   BinarySection *Section{nullptr};
 
   /// Start address of this symbol.
   uint64_t Address{0};
   /// Size of this data (can be 0).
   uint64_t Size{0};
+  /// Output size of this data (if changed).
+  uint64_t OutputSize{0};
   /// Alignment of this data.
   uint16_t Alignment{1};
 
   bool IsMoveable{true};
 
   /// Symbol flags (same as llvm::SymbolRef::Flags)
   unsigned Flags{0};
 
public:
   const BinarySection &getSection() const { return *Section; }
   StringRef getSectionName() const;
 
   BinarySection &getOutputSection() { return *OutputSection; }
   const BinarySection &getOutputSection() const { return *OutputSection; }
   StringRef getOutputSectionName() const;
   uint64_t getOutputAddress() const;
   uint64_t getOutputOffset() const { return OutputOffset; }
-  uint64_t getOutputSize() const { return Size; }
+  uint64_t getOutputSize() const { return OutputSize ? OutputSize : Size; }
+  void setOutputSize(uint64_t NewSize) { OutputSize = NewSize; }
 
   bool isMoved() const;
   bool containsAddress(uint64_t Address) const {
     return ((getAddress() <= Address && Address < getEndAddress()) ||
             (getAddress() == Address && !getSize()));
   }
   bool containsRange(uint64_t Address, uint64_t Size) const {
     return containsAddress(Address) && Address + Size <= getEndAddress();

bolt/include/bolt/Core/BinaryFunction.h

 
 enum IndirectCallPromotionType : char {
   ICP_NONE,        /// Don't perform ICP.
   ICP_CALLS,       /// Perform ICP on indirect calls.
   ICP_JUMP_TABLES, /// Perform ICP on jump tables.
   ICP_ALL          /// Perform ICP on calls and jump tables.
 };
 
+enum IndexOrder {
+  GO_FIRST_BF_INDEX = 0,
+  FIRST_BF_INDEX = 1,
+  GO_UNUSED_BF_INDEX = -3U,
+  GO_LAST_BF_INDEX = -2U,
+  INVALID_BF_INDEX = -1U,
+};
+
 /// Information on a single indirect call to a particular callee.
 struct IndirectCallProfile {
   MCSymbol *Symbol;
   uint32_t Offset;
   uint64_t Count;
   uint64_t Mispreds;
 
   IndirectCallProfile(MCSymbol *Symbol, uint64_t Count, uint64_t Mispreds,
public:
 
   /// List of DWARF CFI instructions. Original CFI from the binary must be
   /// sorted w.r.t. offset that it appears. We rely on this to replay CFIs
   /// if needed (to fix state after reordering BBs).
   using CFIInstrMapType = SmallVector<MCCFIInstruction, 0>;
   using cfi_iterator = CFIInstrMapType::iterator;
   using const_cfi_iterator = CFIInstrMapType::const_iterator;
 
+  // If function is golang stores the offset to functab entry
+  uint32_t GolangFunctabOffset = 0;
+
 private:
   /// Current state of the function.
   State CurrentState{State::Empty};
 
   /// A list of symbols associated with the function entry point.
   ///
   /// Multiple symbols would typically result from identical code-folding
   /// optimization.
private:
 
   /// Size of the function in the output file.
   uint64_t OutputSize{0};
 
   /// Maximum size this function is allowed to have.
   uint64_t MaxSize{std::numeric_limits<uint64_t>::max()};
 
   /// Alignment requirements for the function.
-  uint16_t Alignment{2};
+  uint16_t Alignment{MinAlign};
 
   /// Maximum number of bytes used for alignment of hot part of the function.
   uint16_t MaxAlignmentBytes{0};
 
   /// Maximum number of bytes used for alignment of cold part of the function.
   uint16_t MaxColdAlignmentBytes{0};
 
   const MCSymbol *PersonalityFunction{nullptr};
private:
   /// Set to false by SCTC. Dynostats can't be reliably computed for
   /// functions with non-canonical CFG.
   /// This attribute is only valid when hasCFG() == true.
   bool HasCanonicalCFG{true};
 
   /// True if another function body was merged into this one.
   bool HasFunctionsFoldedInto{false};
 
+  /// True if we have a suspicion that the function was written in ASM
+  bool IsAsm{false};
+
   /// Name for the section this function code should reside in.
   std::string CodeSectionName;
 
   /// Name for the corresponding cold code section.
   std::string ColdCodeSectionName;
 
   /// Parent function fragment for split function fragments.
   SmallPtrSet<BinaryFunction *, 1> ParentFragments;
private:
   /// different parameters by every pass.
   mutable uint64_t Hash{0};
 
   /// For PLT functions it contains a symbol associated with a function
   /// reference. It is nullptr for non-PLT functions.
   const MCSymbol *PLTSymbol{nullptr};
 
   /// Function order for streaming into the destination binary.
-  uint32_t Index{-1U};
+  uint32_t Index{INVALID_BF_INDEX};
 
   /// Get basic block index assuming it belongs to this function.
   unsigned getIndex(const BinaryBasicBlock *BB) const {
     assert(BB->getIndex() < BasicBlocks.size());
     return BB->getIndex();
   }
 
   /// Return basic block that originally contained offset \p Offset
public:
 
   /// Return original address of the function (or offset from base for PIC).
   uint64_t getAddress() const { return Address; }
 
   uint64_t getOutputAddress() const { return OutputAddress; }
 
   uint64_t getOutputSize() const { return OutputSize; }
 
-  /// Does this function have a valid streaming order index?
-  bool hasValidIndex() const { return Index != -1U; }
+  /// Returns true if this function has a valid streaming order index
+  bool hasValidIndex() const { return Index != INVALID_BF_INDEX; }
 
   /// Get the streaming order index for this function.
   uint32_t getIndex() const { return Index; }
 
   /// Set the streaming order index for this function.
   void setIndex(uint32_t Idx) {
-    assert(!hasValidIndex());
     Index = Idx;
   }
 
   /// Return offset of the function body in the binary file.
   uint64_t getFileOffset() const {
     return getLayout().getMainFragment().getFileOffset();
   }
 
public:
   /// fragment.
   bool hasIndirectTargetToSplitFragment() const {
     return HasIndirectTargetToSplitFragment;
   }
 
   /// Return true if all CFG edges have local successors.
   bool hasCanonicalCFG() const { return HasCanonicalCFG; }
 
+  /// Return true if we have a suspicion that the function was written in ASM
+  bool isAsm() const { return IsAsm; }
+
   /// Return true if the original function code has all necessary relocations
   /// to track addresses of functions emitted to new locations.
   bool hasExternalRefRelocations() const { return HasExternalRefRelocations; }
 
   /// Return true if the function has instruction(s) with unknown control flow.
   bool hasUnknownControlFlow() const { return HasUnknownControlFlow; }
 
   /// Return true if the function body is non-contiguous.
public:
   bool hasMemoryProfile() const { return HasMemoryProfile; }
 
   /// Return true if the body of the function was merged into another function.
   bool isFolded() const { return FoldedIntoFunction != nullptr; }
 
   /// Return true if other functions were folded into this one.
   bool hasFunctionsFoldedInto() const { return HasFunctionsFoldedInto; }
 
+  /// Sets golang funtab offset for this function
+  void setGolangFunctabOffset(uint32_t Offset) { GolangFunctabOffset = Offset; }
+
+  /// Return true if the function is presented in original go functab
+  bool isGolang() const { return GolangFunctabOffset != 0; }
+
+  /// Get golang funtab offset for this function
+  uint32_t getGolangFunctabOffset() const { return GolangFunctabOffset; }
+
   /// If this function was folded, return the function it was folded into.
   BinaryFunction *getFoldedIntoFunction() const { return FoldedIntoFunction; }
 
   /// Return true if the function uses jump tables.
   bool hasJumpTables() const { return !JumpTables.empty(); }
 
   /// Return true if the function has SDT marker
   bool hasSDTMarker() const { return HasSDTMarker; }
public:
   void setIsPatched(bool V) { IsPatched = V; }
 
   void setHasIndirectTargetToSplitFragment(bool V) {
     HasIndirectTargetToSplitFragment = V;
   }
 
   void setHasCanonicalCFG(bool V) { HasCanonicalCFG = V; }
 
+  void setIsAsm(bool V) { IsAsm = V; }
+
   void setFolded(BinaryFunction *BF) { FoldedIntoFunction = BF; }
 
   /// Indicate that another function body was merged with this function.
   void setHasFunctionsFoldedInto() { HasFunctionsFoldedInto = true; }
 
   BinaryFunction &setPersonalityFunction(uint64_t Addr) {
     assert(!PersonalityFunction && "can't set personality function twice");
     PersonalityFunction = BC.getOrCreateGlobalSymbol(Addr, "FUNCat");

   }
 
   size_t estimateColdSize() const {
     if (!isSplit())
       return estimateSize();
     size_t Estimate = 0;
     for (const BinaryBasicBlock &BB : blocks())
       if (BB.isCold())
-        Estimate += BC.computeCodeSize(BB.begin(), BB.end());
+        Estimate += BB.estimateSize();
     return Estimate;
   }
 
   size_t estimateSize() const {
     size_t Estimate = 0;
     for (const BinaryBasicBlock &BB : blocks())
-      Estimate += BC.computeCodeSize(BB.begin(), BB.end());
+      Estimate += BB.estimateSize();
     return Estimate;
   }
 
   /// Return output address ranges for a function.
   DebugAddressRangesVector getOutputAddressRanges() const;
 
   /// Given an address corresponding to an instruction in the input binary,
   /// return an address of this instruction in output binary.

bolt/include/bolt/Core/BinarySection.h

 #include "llvm/BinaryFormat/ELF.h"
 #include "llvm/Object/ELFObjectFile.h"
 #include "llvm/Object/MachO.h"
 #include "llvm/Object/ObjectFile.h"
 #include "llvm/Support/raw_ostream.h"
 #include <map>
 #include <memory>
 #include <set>
+#include <shared_mutex>
 
 namespace llvm {
 class MCStreamer;
 class MCSymbol;
 
 using namespace object;
 
 namespace bolt {
class BinarySection {
   uint64_t OutputSize{0};          // Section size in the rewritten binary.
   uint64_t OutputFileOffset{0};    // File offset in the rewritten binary file.
   StringRef OutputContents;        // Rewritten section contents.
   const uint64_t SectionNumber;    // Order in which the section was created.
   unsigned SectionID{-1u};         // Unique ID used for address mapping.
                                    // Set by ExecutableFileMemoryManager.
   uint32_t Index{0};               // Section index in the output file.
   mutable bool IsReordered{false}; // Have the contents been reordered?
+  mutable bool IsChanged{false};   // OutputContents is changed, but the same
+                                   // size as Contents
   bool IsAnonymous{false};         // True if the name should not be included
                                    // in the output file.
 
   uint64_t hash(const BinaryData &BD,
                 std::map<const BinaryData *, uint64_t> &Cache) const;
 
   // non-copyable
   BinarySection(const BinarySection &) = delete;
bool isAllocatable() const {
     if (isELF()) {
       return (ELFFlags & ELF::SHF_ALLOC) && !isTBSS();
     } else {
       // On non-ELF assume all sections are allocatable.
       return true;
     }
   }
   bool isReordered() const { return IsReordered; }
+  bool isChanged() const { return IsChanged; }
   bool isAnonymous() const { return IsAnonymous; }
   unsigned getELFType() const { return ELFType; }
   unsigned getELFFlags() const { return ELFFlags; }
 
   uint8_t *getData() {
     return reinterpret_cast<uint8_t *>(
         const_cast<char *>(getContents().data()));
   }
public:
   }
 
   /// Does this section have any non-pending relocations?
   bool hasRelocations() const { return !Relocations.empty(); }
 
   /// Does this section have any pending relocations?
   bool hasPendingRelocations() const { return !PendingRelocations.empty(); }
 
+  /// Does this section have any dynamic relocations?
+  bool hasDynamicRelocations() const { return !DynamicRelocations.empty(); }
+
   /// Remove non-pending relocation with the given /p Offset.
   bool removeRelocationAt(uint64_t Offset) {
     auto Itr = Relocations.find(Offset);
     if (Itr != Relocations.end()) {
       Relocations.erase(Itr);
       return true;
     }
+
+    return false;
+  }
+
+  /// Remove the dynamic relocation (if any) at the given /p Offset.
+  bool removeDynamicRelocationAt(uint64_t Offset) {
+    Relocation Key{Offset, 0, 0, 0, 0};
+    auto Itr = DynamicRelocations.find(Key);
+    if (Itr != DynamicRelocations.end()) {
+      DynamicRelocations.erase(Itr);
+      return true;
+    }
+
     return false;
   }
 
   void clearRelocations();
 
   /// Add a new relocation at the given /p Offset.
   void addRelocation(uint64_t Offset, MCSymbol *Symbol, uint64_t Type,
-                     uint64_t Addend, uint64_t Value = 0,
+                     uint64_t Addend = 0, uint64_t Value = 0,
                      bool Pending = false) {
     assert(Offset < getSize() && "offset not within section bounds");
     if (!Pending) {
       Relocations.emplace(Relocation{Offset, Symbol, Type, Addend, Value});
     } else {
       PendingRelocations.emplace_back(
           Relocation{Offset, Symbol, Type, Addend, Value});
     }
   }
 
   /// Add a dynamic relocation at the given /p Offset.
   void addDynamicRelocation(uint64_t Offset, MCSymbol *Symbol, uint64_t Type,
-                            uint64_t Addend, uint64_t Value = 0) {
+                            uint64_t Addend = 0, uint64_t Value = 0) {
     assert(Offset < getSize() && "offset not within section bounds");
     DynamicRelocations.emplace(Relocation{Offset, Symbol, Type, Addend, Value});
   }
 
   /// Add relocation against the original contents of this section.
   void addPendingRelocation(const Relocation &Rel) {
     PendingRelocations.push_back(Rel);
   }
public:
   BinaryPatcher *getPatcher() { return Patcher.get(); }
 
   /// Lookup the relocation (if any) at the given /p Offset.
   const Relocation *getRelocationAt(uint64_t Offset) const {
     auto Itr = Relocations.find(Offset);
     return Itr != Relocations.end() ? &*Itr : nullptr;
   }
 
-  /// Lookup the relocation (if any) at the given /p Offset.
+  /// Lookup the dynamic relocation (if any) at the given /p Offset.
   const Relocation *getDynamicRelocationAt(uint64_t Offset) const {
     Relocation Key{Offset, 0, 0, 0, 0};
     auto Itr = DynamicRelocations.find(Key);
     return Itr != DynamicRelocations.end() ? &*Itr : nullptr;
   }
 
   uint64_t hash(const BinaryData &BD) const {
     std::map<const BinaryData *, uint64_t> Cache;
public:
   uint64_t getOutputSize() const { return OutputSize; }
   uint8_t *getOutputData() {
     return reinterpret_cast<uint8_t *>(
         const_cast<char *>(getOutputContents().data()));
   }
   const uint8_t *getOutputData() const {
     return reinterpret_cast<const uint8_t *>(getOutputContents().data());
   }
   StringRef getOutputContents() const { return OutputContents; }
   uint64_t getAllocAddress() const {
     return reinterpret_cast<uint64_t>(getOutputData());
   }
   uint64_t getOutputAddress() const { return OutputAddress; }
   uint64_t getOutputFileOffset() const { return OutputFileOffset; }
   unsigned getSectionID() const {
     assert(hasValidSectionID() && "trying to use uninitialized section id");
     return SectionID;
public:
   void flushPendingRelocations(raw_pwrite_stream &OS,
                                SymbolResolverFuncTy Resolver);
 
   /// Change contents of the section.
   void updateContents(const uint8_t *Data, size_t NewSize) {
     OutputContents = StringRef(reinterpret_cast<const char *>(Data), NewSize);
     OutputSize = NewSize;
     IsFinalized = true;
+    IsChanged = true;
   }
 
   /// Reorder the contents of this section according to /p Order.  If
   /// /p Inplace is true, the entire contents of the section is reordered,
   /// otherwise the new contents contain only the reordered data.
   void reorderContents(const std::vector<BinaryData *> &Order, bool Inplace);
 
   void print(raw_ostream &OS) const;

bolt/include/bolt/Core/MCPlusBuilder.h

private:
     std::unique_ptr<Impl> Itr;
   };
 
 public:
   MCPlusBuilder(const MCInstrAnalysis *Analysis, const MCInstrInfo *Info,
                 const MCRegisterInfo *RegInfo)
       : Analysis(Analysis), Info(Info), RegInfo(RegInfo) {
     // Initialize the default annotation allocator with id 0
-    AnnotationAllocators.emplace(0, AnnotationAllocator());
-    MaxAllocatorId++;
+    AnnotationAllocators.emplace(MaxAllocatorId++, AnnotationAllocator());
     // Build alias map
     initAliases();
     initSizeMap();
   }
 
   /// Create and return target-specific MC symbolizer for the \p Function.
   virtual std::unique_ptr<MCSymbolizer>
   createTargetSymbolizer(BinaryFunction &Function) const {
public:
 
   /// Convert a stack accessing load/store instruction in \p Inst to a PUSH
   /// or POP saving/restoring the source/dest reg in \p Inst. The original
   /// stack offset in \p Inst is ignored.
   virtual void changeToPushOrPop(MCInst &Inst) const {
     llvm_unreachable("not implemented");
   }
 
+  /// Get stack adjustment value
+  virtual int getStackAdjustment(const MCInst &Inst) const {
+    llvm_unreachable("not implemented");
+    return false;
+  }
+
   /// Identify stack adjustment instructions -- those that change the stack
   /// pointer by adding or subtracting an immediate.
   virtual bool isStackAdjustment(const MCInst &Inst) const {
     llvm_unreachable("not implemented");
     return false;
   }
 
   /// Use \p Input1 or Input2 as the current value for the input

   virtual bool
   convertMoveToConditionalMove(MCInst &Inst, unsigned CC,
                                bool AllowStackMemOp = false,
                                bool AllowBasePtrStackMemOp = false) const {
     llvm_unreachable("not implemented");
     return false;
   }
 
+  /// Replace instruction with relaxed version of it
+  virtual bool relaxInstruction(MCInst &Inst) const {
+    llvm_unreachable("not implemented");
+    return false;
+  }
+
   /// Lower a tail call instruction \p Inst if required by target.
   virtual bool lowerTailCall(MCInst &Inst) {
     llvm_unreachable("not implemented");
     return false;
   }
 
   /// Receives a list of MCInst of the basic block to analyze and interpret the
   /// terminators of this basic block. TBB must be initialized with the original
unsigned getOrCreateAnnotationIndex(StringRef Name) {
 
     const unsigned Index =
         AnnotationNameIndexMap.size() + MCPlus::MCAnnotation::kGeneric;
     AnnotationNameIndexMap.insert(std::make_pair(Name, Index));
     AnnotationNames.emplace_back(std::string(Name));
     return Index;
   }
 
+  /// Copy annotations from one instruction to other
+  void copyAnnotationInst(const MCInst &From, MCInst &To,
+                          AllocatorIdTy AllocatorId = 0) {
+    const MCInst *AnnotationInst = getAnnotationInst(From);
+    if (!AnnotationInst)
+      return;
+
+    for (unsigned I = 0; I < AnnotationInst->getNumOperands(); ++I) {
+      const int64_t Imm = AnnotationInst->getOperand(I).getImm();
+      const unsigned Index = extractAnnotationIndex(Imm);
+      const int64_t Value = extractAnnotationValue(Imm);
+      setAnnotationOpValue(To, Index, Value, AllocatorId);
+    }
+  }
+
+  /// Comapre annotations stored in instructions and return true if equal.
+  /// If \p GenericOnly is true compare only generic annotations.
+  bool areAnnotationsEqual(const MCInst &InstA, const MCInst &InstB,
+                           bool GenericOnly = true) {
+    const MCInst *AnnotationInstA = getAnnotationInst(InstA);
+    const MCInst *AnnotationInstB = getAnnotationInst(InstB);
+    if (!!AnnotationInstA != !!AnnotationInstB)
+      return false;
+
+    if (!AnnotationInstA)
+      return true;
+
+    if (AnnotationInstA->getNumOperands() != AnnotationInstB->getNumOperands())
+      return false;
+
+    for (unsigned I = 0; I < AnnotationInstA->getNumOperands(); ++I) {
+      const int64_t ImmA = AnnotationInstA->getOperand(I).getImm();
+      const unsigned Index = extractAnnotationIndex(ImmA);
+      const int64_t ValueA = extractAnnotationValue(ImmA);
+      const auto ValueB = getAnnotationOpValue(InstB, Index);
+      if (!ValueB)
+        return false;
+
+      if (Index >= MCPlus::MCAnnotation::kGeneric) {
+        const auto *AnnotationA =
+            reinterpret_cast<const MCPlus::MCAnnotation *>(ValueA);
+        const auto *AnnotationB =
+            reinterpret_cast<const MCPlus::MCAnnotation *>(*ValueB);
+        if (!AnnotationA->equals(*AnnotationB))
+          return false;
+      } else if (!GenericOnly && ValueA != *ValueB) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
   /// Store an annotation value on an MCInst.  This assumes the annotation
   /// is not already present.
   template <typename ValueType>
   const ValueType &addAnnotation(MCInst &Inst, unsigned Index,
                                  const ValueType &Val,
                                  AllocatorIdTy AllocatorId = 0) {
     assert(!hasAnnotation(Inst, Index));
     AnnotationAllocator &Allocator = getAnnotationAllocator(AllocatorId);

   virtual InstructionListType
   createInstrumentedIndCallHandlerEntryBB(const MCSymbol *InstrTrampoline,
                                           const MCSymbol *IndCallHandler,
                                           MCContext *Ctx) {
     llvm_unreachable("not implemented");
     return InstructionListType();
   }
 
+  virtual InstructionListType
+  createInstrumentFiniCall(MCSymbol *HandlerFuncAddr, MCContext *Ctx,
+                           bool IsTailCall) {
+    llvm_unreachable("not implemented");
+    return std::vector<MCInst>();
+  }
+
   virtual InstructionListType createNumCountersGetter(MCContext *Ctx) const {
     llvm_unreachable("not implemented");
     return {};
   }
 
   virtual InstructionListType createInstrLocationsGetter(MCContext *Ctx) const {
     llvm_unreachable("not implemented");
     return {};

bolt/include/bolt/Core/Relocation.h

struct Relocation {
   static uint64_t getNone();
 
   /// Return code for a PC-relative 4-byte relocation
   static uint64_t getPC32();
 
   /// Return code for a PC-relative 8-byte relocation
   static uint64_t getPC64();
 
+  /// Return code for a ABS 4-byte relocation
+  static uint64_t getAbs32();
+
   /// Return code for a ABS 8-byte relocation
   static uint64_t getAbs64();
 
+  /// Return code for ABS relocation based on size
+  static uint64_t getAbs(uint8_t Size);
+
   /// Return code for a RELATIVE relocation
   static uint64_t getRelative();
 
   /// Return true if this relocation is PC-relative. Return false otherwise.
   bool isPCRelative() const { return isPCRelative(Type); }
 
   /// Return true if this relocation is R_*_RELATIVE type. Return false
   /// otherwise.
   bool isRelative() const { return isRelative(Type); }
 
+  /// Return true if this relocation is IRELATIVE type. Return false otherwise.
+  bool isIRelative() const { return isIRelative(Type); }
+
   /// Emit relocation at a current \p Streamer' position. The caller is
   /// responsible for setting the position correctly.
   size_t emit(MCStreamer *Streamer) const;
 
   /// Print a relocation to \p OS.
   void print(raw_ostream &OS) const;
 };
 

bolt/include/bolt/Passes/Golang.h

+//===--------- Passes/Golang.h ----------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Golang binaries support passes.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVM_BOLT_PASSES_GOLANG_H
+#define LLVM_TOOLS_LLVM_BOLT_PASSES_GOLANG_H
+
+#include "BinaryPasses.h"
+#include "bolt/Utils/CommandLineOpts.h"
+
+#include "Golang/go_v1_14.h"
+#include "Golang/go_v1_16_5.h"
+#include "Golang/go_v1_17.h"
+
+namespace llvm {
+namespace bolt {
+
+class GolangPass : public BinaryFunctionPass {
+protected:
+  uint64_t RuntimeText = 0;
+  std::unique_ptr<struct Module> FirstModule;
+  std::unique_ptr<class Pclntab> Pclntab;
+
+public:
+  /// Golang version strings
+  const char *GolangStringVer[opts::GV_LAST] = {
+      "none",     "auto",     "go1.14.9", "go1.14.12",
+      "go1.16.5", "go1.17.2", "go1.17.5", "go1.17.8",
+  };
+
+  explicit GolangPass(BinaryContext &BC) : BinaryFunctionPass(false) {
+    if (checkGoVersion(BC) < 0) {
+      errs() << "BOLT-ERROR: Failed to check golang version!\n";
+      exit(1);
+    }
+
+    if (getSymbols(BC) < 0) {
+      errs() << "BOLT-ERROR: Failed to get golang-specific symbols!\n";
+      exit(1);
+    }
+
+    FirstModule = createGoModule();
+    if (!FirstModule || FirstModule->read(BC) < 0) {
+      errs() << "BOLT-ERROR: Failed to read firstmodule!\n";
+      exit(1);
+    }
+
+    Pclntab = createGoPclntab();
+    if (!Pclntab || Pclntab->readHeader(BC, getPcHeaderAddr()) < 0) {
+      errs() << "BOLT-ERROR: Failed to read pclntab!\n";
+      exit(1);
+    }
+
+    // NOTE last entry is etext
+    if (Pclntab->getFunctionsCount() != FirstModule->getFtab().getCount() - 1) {
+      errs() << "BOLT-ERROR: Wrong symtab size!\n";
+      exit(1);
+    }
+  }
+
+  static const char *getFirstBFName(void) {
+    const char *const Name = "runtime.text";
+    return Name;
+  }
+
+  static const char *getLastBFName(void) {
+    const char *const Name = "runtime.etext";
+    return Name;
+  }
+
+  static uint32_t getUndAarch64(void) { return 0xbea71700; }
+
+  uint64_t getPcHeaderAddr() const { return FirstModule->getPcHeaderAddr(); }
+
+  uint8_t getPsize() const { return Pclntab->getPsize(); }
+
+  const class Pclntab *getPclntab() const {
+    return const_cast<class Pclntab *>(Pclntab.get());
+  }
+
+  const char *getName() const override { return "golang"; }
+
+  /// Pass entry point
+  void runOnFunctions(BinaryContext &BC) override;
+  int checkGoVersion(BinaryContext &BC);
+  int getSymbols(BinaryContext &BC);
+  int textsectmapPass(BinaryContext &BC);
+  int typePass(BinaryContext &BC, uint64_t TypeAddr);
+  int typelinksPass(BinaryContext &BC);
+  int pcspPass(BinaryFunction *BF, uint8_t **SectionData, const uint32_t Index,
+               uint8_t Quantum, bool ForcePreserve);
+  uint32_t deferreturnPass(BinaryContext &BC, BinaryFunction *BF);
+  int getNextMCinstVal(FunctionLayout::block_iterator BBIt, uint64_t I,
+                       const uint32_t Index, int32_t &Val,
+                       uint64_t *NextOffset);
+  int writeVarintPass(BinaryFunction *BF, uint8_t **DataFuncOffset,
+                      const uint32_t Index, const uint8_t Quantum);
+  int unsafePointPass(BinaryFunction *BF, GoFunc *GoFunc);
+  int pclntabPass(BinaryContext &BC);
+  int findFuncTabPass(BinaryContext &BC);
+};
+
+class GolangPostPass : public GolangPass {
+public:
+  explicit GolangPostPass(BinaryContext &BC) : GolangPass(BC) {}
+
+  /// Pass entry point
+  void runOnFunctions(BinaryContext &BC) override;
+  void skipPleaseUseCallersFramesPass(BinaryContext &BC);
+  void instrumentExitCall(BinaryContext &BC);
+  uint32_t pcdataPass(BinaryFunction *BF, GoFunc *GoFunc, const uint32_t Index,
+                      const unsigned AllocId);
+  int pclntabPass(BinaryContext &BC);
+};
+
+class GolangPrePass : public GolangPass {
+public:
+  explicit GolangPrePass(BinaryContext &BC) : GolangPass(BC) {}
+
+  /// Pass entry point
+  void runOnFunctions(BinaryContext &BC) override;
+  void goPassInit(BinaryContext &BC);
+  void nopPass(BinaryContext &BC);
+  int pclntabPass(BinaryContext &BC);
+  void deferreturnPass(BinaryFunction &BF, const uint64_t DeferOffset,
+                       const unsigned AllocId, const MCCodeEmitter *Emitter);
+};
+
+} // namespace bolt
+} // namespace llvm
+
+#endif

bolt/include/bolt/Passes/Golang/go_base.h

+//===--------- Passes/Golang/go_base.h --------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVM_BOLT_GOLANG_BASE_H
+#define LLVM_TOOLS_LLVM_BOLT_GOLANG_BASE_H
+
+#include "go_common.h"
+
+namespace llvm {
+namespace bolt {
+
+// runtime/symtab.go
+struct Functab {
+  uint64_t Address; // Pointer size
+  uint64_t Offset;  // Pointer size
+};
+
+// runtime/symtab.go
+struct InlinedCall {
+  int16_t Parent;
+  uint8_t FuncID;
+  uint8_t Unused;
+  uint32_t File;
+  uint32_t Line;
+  uint32_t Func;
+  uint32_t ParentPc;
+};
+
+class GoArray {
+public:
+  uint64_t Address;
+  uint64_t Count[2];
+
+  uint64_t getAddress() const { return Address; }
+
+  void setAddress(uint64_t Addr) { Address = Addr; }
+
+  uint64_t getCount() const { return Count[0]; }
+
+  void setCount(uint64_t C) {
+    Count[0] = C;
+    Count[1] = C;
+  }
+};
+
+// moduledata struct
+// runtime/symtab.go
+// NOTE: Every field size is target-machines pointer size
+// NOTE: For some reason array[] fields count is repeated in struct
+struct Module {
+  virtual ~Module() = 0;
+
+  BinaryData *getModuleBD(BinaryContext &BC) {
+    BinaryData *Module = BC.getFirstBinaryDataByName("local.moduledata");
+    if (!Module)
+      Module = BC.getFirstBinaryDataByName("runtime.firstmoduledata");
+
+    return Module;
+  }
+
+  int read(const BinaryContext &BC);
+  virtual uint64_t getFieldOffset(BinaryContext &BC, uint64_t *Addr) const = 0;
+  virtual int patch(BinaryContext &BC) = 0;
+  virtual uint64_t *getModule() = 0;
+  virtual size_t getModuleSize() const = 0;
+  virtual void setPclntabSize(uint64_t Size) = 0;
+  virtual void setFtabSize(uint64_t Count) = 0;
+  virtual uint64_t getPcHeaderAddr() const = 0;
+  virtual const GoArray &getFtab() const = 0;
+  virtual uint64_t getFindfunctab() const = 0;
+  virtual uint64_t getTypes() const = 0;
+  virtual uint64_t getEtypes() const = 0;
+  virtual const GoArray &getTextsectmap() const = 0;
+  virtual const GoArray &getTypelinks() const = 0;
+};
+
+std::unique_ptr<struct Module> createGoModule();
+
+// runtime/symtab.go
+class Pclntab {
+  uint64_t PclntabHeaderOffset = 0;
+
+  virtual void __readHeader(BinaryContext &BC, DataExtractor &DE) = 0;
+  virtual void __writeHeader(BinaryContext &BC, uint8_t *Pclntab) const = 0;
+  virtual bool checkMagic() const = 0;
+  virtual void setNewHeaderOffsets() = 0;
+
+protected:
+  void setPclntabHeaderOffset(uint64_t Off) { PclntabHeaderOffset = Off; }
+
+  uint64_t getPclntabHeaderOffset() const { return PclntabHeaderOffset; }
+
+public:
+  virtual ~Pclntab() = 0;
+  int readHeader(BinaryContext &BC, const uint64_t PclntabHeaderAddr);
+  int writeHeader(BinaryContext &BC, uint8_t *Pclntab);
+  virtual size_t getPcHeaderSize() const = 0;
+  virtual void setFunctionsCount(uint64_t Count) = 0;
+  virtual uint8_t getQuantum() const = 0;
+  virtual uint8_t getPsize() const = 0;
+  virtual uint64_t getFunctionsCount() const = 0;
+  virtual uint64_t getNameOffset() const = 0;
+  virtual uint64_t getFiletabOffset() const = 0;
+  virtual uint64_t getPctabOffset() const = 0;
+  virtual uint64_t getPclntabOffset() const = 0;
+  virtual uint64_t getFunctabOffset() const = 0;
+};
+
+std::unique_ptr<class Pclntab> createGoPclntab();
+
+// runtime/runtime2.go
+struct GoFunc {
+  virtual ~GoFunc() = 0;
+
+  uint64_t PcdataOffset = 0;
+  uint64_t FuncdataOffset = 0;
+
+  virtual void __read(const BinaryContext &BC, DataExtractor &DE,
+                      BinarySection *Section, uint64_t *FuncOffset) = 0;
+
+  virtual void __write(BinaryFunction *BF, uint8_t **FuncPart, uint8_t *Section,
+                       BinarySection *OutputSection) const = 0;
+
+  virtual size_t getSize(BinaryContext &BC) const = 0;
+
+  int read(const BinaryContext &BC, DataExtractor &DE, BinarySection *Section,
+           uint64_t *FuncOffset) {
+    __read(BC, DE, Section, FuncOffset);
+
+    // Read pcdata
+    PcdataOffset = *FuncOffset;
+    for (uint32_t I = 0; I < getNpcdata(); ++I)
+      setPcdata(I, (uint32_t)readEndianVal(DE, FuncOffset, sizeof(uint32_t)));
+
+    // Read funcdata
+    *FuncOffset = alignTo(*FuncOffset, BC.AsmInfo->getCodePointerSize());
+    FuncdataOffset = *FuncOffset;
+    for (uint32_t I = 0; I < getNfuncdata(); ++I)
+      setFuncdata(
+          I, readEndianVal(DE, FuncOffset, BC.AsmInfo->getCodePointerSize()));
+
+    return 0;
+  }
+
+  int read(const BinaryFunction &BF) {
+    if (!BF.isGolang())
+      return -1;
+
+    const BinaryContext &BC = BF.getBinaryContext();
+    std::unique_ptr<struct Module> FirstModule = createGoModule();
+    FirstModule->read(BC);
+
+    const uint64_t PclntabAddr = FirstModule->getPcHeaderAddr();
+    if (!PclntabAddr) {
+      errs() << "BOLT-ERROR: Pclntab address is zero!\n";
+      return -1;
+    }
+
+    const BinaryData *PclntabSym = BC.getBinaryDataAtAddress(PclntabAddr);
+    if (!PclntabSym) {
+      errs() << "BOLT-ERROR: Failed to get pclntab symbol!\n";
+      return -1;
+    }
+
+    const BinarySection *Section = &PclntabSym->getSection();
+    DataExtractor DE =
+        DataExtractor(Section->getContents(), BC.AsmInfo->isLittleEndian(),
+                      BC.AsmInfo->getCodePointerSize());
+    uint64_t FuncOffset = BF.getGolangFunctabOffset();
+    return read(BC, DE, nullptr, &FuncOffset);
+  }
+
+  int write(BinaryFunction *BF, uint8_t **FuncPart, uint8_t *SectionData,
+            BinarySection *OutputSection) {
+    BinaryContext &BC = BF->getBinaryContext();
+    __write(BF, FuncPart, SectionData, OutputSection);
+
+    // Write pcdata
+    for (uint32_t I = 0; I < getNpcdata(); ++I)
+      writeEndianVal(BC, getPcdata(I), sizeof(uint32_t), FuncPart);
+
+    // Write funcdata
+    *FuncPart = SectionData + alignTo(*FuncPart - SectionData,
+                                      BC.AsmInfo->getCodePointerSize());
+    for (uint32_t I = 0; I < getNfuncdata(); ++I) {
+      uint64_t Val = getFuncdata(I);
+      if (Val) {
+        uint64_t Delta = (uint64_t)(*FuncPart - SectionData);
+        AddRelaReloc(BC, nullptr, OutputSection, Delta, Val);
+        *FuncPart += BC.AsmInfo->getCodePointerSize();
+      } else {
+        writeEndianPointer(BC, 0, FuncPart);
+      }
+    }
+    return 0;
+  }
+
+  virtual void disableMetadata() = 0;
+  virtual int32_t getNameOffset() const = 0;
+  virtual void setNameOffset(int32_t Offset) = 0;
+  virtual uint32_t getDeferreturnOffset() const = 0;
+  virtual void setDeferreturnOffset(uint32_t Offset) = 0;
+  virtual uint32_t getPcspOffset() const = 0;
+  virtual void setPcspOffset(uint32_t Offset) = 0;
+  virtual uint32_t getNpcdata() const = 0;
+  virtual void fixNpcdata() = 0;
+  virtual bool hasReservedID(std::string Name) const = 0;
+  virtual uint8_t getNfuncdata() const = 0;
+
+  // runtime/symtab.go
+  virtual unsigned getPcdataUnsafePointIndex() const = 0;
+  virtual unsigned getPcdataStackMapIndex() const = 0;
+  virtual unsigned getPcdataInlTreeIndex() const = 0;
+  virtual unsigned getPcdataMaxIndex() const = 0;
+  virtual size_t getPcdataSize() const = 0;
+  virtual uint32_t getPcdata(unsigned Index) const = 0;
+  virtual void setPcdata(unsigned Index, uint32_t Value) = 0;
+  virtual void setPcdataMaxVal(unsigned Index, int32_t Value) = 0;
+  virtual int32_t getPcdataMax(unsigned Index) const = 0;
+  virtual int getPcdataSafePointVal() const = 0;
+  virtual int getPcdataUnsafePointVal() const = 0;
+  virtual unsigned getFuncdataInlTreeIndex() const = 0;
+  virtual uint64_t getFuncdata(unsigned Index) const = 0;
+  virtual void setFuncdata(unsigned Index, uint64_t Value) = 0;
+
+  uint32_t getPcdataOffset() const { return PcdataOffset; }
+
+  uint32_t getFuncdataOffset() const { return FuncdataOffset; }
+
+  // Fake index used to store pcsp values
+  unsigned getPcspIndex() const { return getPcdataMaxIndex() + 1; }
+};
+
+std::unique_ptr<struct GoFunc> createGoFunc();
+
+} // namespace bolt
+} // namespace llvm
+
+#endif

bolt/include/bolt/Passes/Golang/go_common.h

+//===--------- Passes/Golang/go_common.h ------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVM_BOLT_GOLANG_COMMON_H
+#define LLVM_TOOLS_LLVM_BOLT_GOLANG_COMMON_H
+
+namespace llvm {
+namespace bolt {
+
+BinaryFunction *getBF(BinaryContext &BC, std::vector<BinaryFunction *> &BFs,
+                      const char *Name);
+
+BinaryFunction *getFirstBF(BinaryContext &BC,
+                           std::vector<BinaryFunction *> &BFs);
+
+BinaryFunction *getLastBF(BinaryContext &BC,
+                          std::vector<BinaryFunction *> &BFs);
+
+void writeEndianVal(BinaryContext &BC, uint64_t Val, uint16_t Size,
+                    uint8_t **Res);
+
+void writeEndianPointer(BinaryContext &BC, uint64_t Val, uint8_t **Res);
+
+inline uint64_t readEndianVal(DataExtractor &DE, uint64_t *Offset,
+                              uint16_t Size) {
+  assert(DE.isValidOffset(*Offset) && "Invalid offset");
+  switch (Size) {
+  case 8:
+    return DE.getU64(Offset);
+  case 4:
+    return DE.getU32(Offset);
+  case 2:
+    return DE.getU16(Offset);
+  case 1:
+    return DE.getU8(Offset);
+  default:
+    __builtin_unreachable();
+  }
+}
+
+inline uint32_t readVarint(const uint8_t *Data, uint64_t *Offset) {
+  uint32_t res = 0, shift = 0;
+  uint8_t val;
+
+  while (1) {
+    val = Data[(*Offset)++];
+    res |= ((uint32_t)(val & 0x7F)) << (shift & 31);
+    if ((val & 0x80) == 0)
+      break;
+
+    shift += 7;
+  }
+
+  return res;
+}
+
+inline int32_t readVarintPair(const uint8_t *Data, uint64_t *Offset,
+                              int32_t &ValSum, uint64_t &OffsetSum,
+                              const uint8_t Quantum) {
+  uint32_t Val = readVarint(Data, Offset);
+  ValSum += (-(Val & 1) ^ (Val >> 1));
+  OffsetSum += readVarint(Data, Offset) * Quantum;
+  return (int32_t)Val;
+}
+
+inline int32_t readVarintPair(DataExtractor &DE, uint64_t *Offset,
+                              int32_t &ValSum, uint64_t &OffsetSum,
+                              const uint8_t Quantum) {
+  assert(DE.isValidOffset(*Offset));
+  const uint8_t *Data = (const uint8_t *)DE.getData().data();
+  return readVarintPair(Data, Offset, ValSum, OffsetSum, Quantum);
+}
+
+inline void AddRelaReloc(BinaryContext &BC, MCSymbol *Symbol,
+                         BinarySection *Section, uint64_t Offset,
+                         uint64_t Addend) {
+  Section->addRelocation(Offset, Symbol,
+                         Relocation::getAbs(BC.AsmInfo->getCodePointerSize()),
+                         Addend);
+
+  if (!BC.HasFixedLoadAddress)
+    Section->addDynamicRelocation(Offset, Symbol, Relocation::getRelative(),
+                                  Addend);
+}
+
+inline void RemoveRelaReloc(const BinaryContext &BC, BinarySection *Section,
+                            uint64_t Offset) {
+  Section->removeRelocationAt(Offset);
+  Section->removeDynamicRelocationAt(Offset);
+}
+
+inline std::string getVarintName(uint32_t Index, bool IsNext = false) {
+  const char *const Varint = "VARINT";
+  const char *const VarintNext = "VARINT_NEXT";
+
+  std::string Name = IsNext ? VarintNext : Varint;
+  Name += std::to_string(Index);
+  return Name;
+}
+
+inline void addVarintAnnotation(BinaryContext &BC, MCInst &II, uint32_t Index,
+                                int32_t Value, bool IsNext,
+                                unsigned AllocId = 0) {
+  BC.MIB->addAnnotation(II, getVarintName(Index, IsNext), Value, AllocId);
+}
+
+inline bool hasVarintAnnotation(BinaryContext &BC, MCInst &II, uint32_t Index,
+                                bool IsNext = false) {
+  return BC.MIB->hasAnnotation(II, getVarintName(Index, IsNext));
+}
+
+inline int32_t getVarintAnnotation(BinaryContext &BC, MCInst &II,
+                                   uint32_t Index, bool IsNext = false) {
+  return BC.MIB->getAnnotationAs<int32_t>(II, getVarintName(Index, IsNext));
+}
+
+inline std::string getFuncdataName(uint32_t Findex, uint32_t Size) {
+  return "FUNCDATA" + std::to_string(Findex) + std::to_string(Size);
+}
+
+inline std::string getFuncdataSizeName(uint32_t Findex) {
+  return "FUNCDATA_SIZE" + std::to_string(Findex);
+}
+
+inline void addFuncdataAnnotation(BinaryContext &BC, MCInst &II,
+                                  uint32_t Findex, int32_t Value,
+                                  unsigned AllocId = 0) {
+  auto &Size = BC.MIB->getOrCreateAnnotationAs<uint32_t>(
+      II, getFuncdataSizeName(Findex), AllocId);
+  BC.MIB->addAnnotation(II, getFuncdataName(Findex, Size++), Value, AllocId);
+}
+
+inline bool hasFuncdataAnnotation(BinaryContext &BC, MCInst &II,
+                                  uint32_t Findex) {
+  return BC.MIB->hasAnnotation(II, getFuncdataSizeName(Findex));
+}
+
+inline uint32_t getFuncdataSizeAnnotation(BinaryContext &BC, MCInst &II,
+                                          uint32_t Findex) {
+  return BC.MIB->getAnnotationAs<uint32_t>(II, getFuncdataSizeName(Findex));
+}
+
+inline int32_t getFuncdataAnnotation(BinaryContext &BC, MCInst &II,
+                                     uint32_t Findex, uint32_t Index) {
+  return BC.MIB->getAnnotationAs<int32_t>(II, getFuncdataName(Findex, Index));
+}
+
+} // namespace bolt
+} // namespace llvm
+
+#endif

bolt/include/bolt/Passes/Golang/go_v1_14.h

+//===--------- Passes/Golang/go_v1_14.h -------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVM_BOLT_GOLANG_V1_14_H
+#define LLVM_TOOLS_LLVM_BOLT_GOLANG_V1_14_H
+
+#include "go_base.h"
+
+namespace llvm {
+namespace bolt {
+
+class Pclntab_v1_14_9 : public Pclntab {
+// runtime/symtab.go moduledataverify1
+#define PclntabFields                                                          \
+  F(uint32_t, false, Magic)                                                    \
+  F(uint8_t, false, Zero1)                                                     \
+  F(uint8_t, false, Zero2)                                                     \
+  F(uint8_t, false, Quantum)                                                   \
+  F(uint8_t, false, Psize)                                                     \
+  F(uint64_t, true, SymtabSize)
+
+  struct PcHeader {
+#define F(Type, IsPointerSize, Field) Type Field;
+    PclntabFields
+#undef F
+  } Header;
+
+  void __readHeader(BinaryContext &BC, DataExtractor &DE) override {
+    uint64_t Offset = getPclntabHeaderOffset();
+#define F(Type, IsPointer, Field)                                              \
+  {                                                                            \
+    assert(DE.isValidOffset(Offset) && "Invalid offset");                      \
+    size_t FSize =                                                             \
+        IsPointer ? BC.AsmInfo->getCodePointerSize() : sizeof(Type);           \
+    Header.Field = readEndianVal(DE, &Offset, FSize);                          \
+  }
+    PclntabFields
+#undef F
+  }
+
+  void __writeHeader(BinaryContext &BC, uint8_t *Pclntab) const override {
+#define F(Type, IsPointer, Field)                                              \
+  {                                                                            \
+    size_t FSize =                                                             \
+        IsPointer ? BC.AsmInfo->getCodePointerSize() : sizeof(Type);           \
+    writeEndianVal(BC, Header.Field, FSize, &Pclntab);                         \
+  }
+    PclntabFields
+#undef F
+  }
+
+  void setNewHeaderOffsets() override {}
+
+  bool checkMagic() const override { return Header.Magic == 0xfffffffb; }
+
+public:
+  ~Pclntab_v1_14_9() = default;
+
+  static size_t getPcHeaderSize(unsigned Psize) {
+    size_t FuncSize = 0;
+#define F(Type, IsPointerSize, Field)                                          \
+  if (IsPointerSize)                                                           \
+    FuncSize += Psize;                                                         \
+  else                                                                         \
+    FuncSize += sizeof(Type);
+    PclntabFields
+#undef F
+
+        return alignTo(FuncSize, Psize);
+  }
+
+#undef PclntabFields
+
+  size_t getPcHeaderSize() const override {
+    return getPcHeaderSize(Header.Psize);
+  }
+
+  void setFunctionsCount(uint64_t Count) override { Header.SymtabSize = Count; }
+
+  uint8_t getQuantum() const override { return Header.Quantum; }
+
+  uint8_t getPsize() const override { return Header.Psize; }
+
+  uint64_t getFunctionsCount() const override { return Header.SymtabSize; }
+
+  uint64_t getNameOffset() const override { return getPclntabHeaderOffset(); }
+
+  uint64_t getFiletabOffset() const override {
+    return getPclntabHeaderOffset();
+  }
+
+  uint64_t getPctabOffset() const override { return getPclntabHeaderOffset(); }
+
+  uint64_t getPclntabOffset() const override {
+    return getPclntabHeaderOffset() + getPcHeaderSize();
+  }
+
+  uint64_t getFunctabOffset() const override {
+    return getPclntabHeaderOffset();
+  }
+};
+
+struct GoFunc_v1_14_9 : GoFunc {
+  ~GoFunc_v1_14_9() = default;
+
+  // runtime/symtab.go
+  enum {
+    _PCDATA_RegMapIndex = 0,
+    _PCDATA_StackMapIndex = 1,
+    _PCDATA_InlTreeIndex = 2,
+    _PCDATA_MAX,
+    _FUNCDATA_ArgsPointerMaps = 0,
+    _FUNCDATA_LocalsPointerMaps = 1,
+    _FUNCDATA_RegPointerMaps = 2,
+    _FUNCDATA_StackObjects = 3,
+    _FUNCDATA_InlTree = 4,
+    _FUNCDATA_OpenCodedDeferInfo = 5,
+    _FUNCDATA_MAX,
+    _ArgsSizeUnknown = -0x80000000
+  };
+
+  // runtime/symtab.go
+  enum {
+    funcID_normal = 0,
+    funcID_runtime_main,
+    funcID_goexit,
+    funcID_jmpdefer,
+    funcID_mcall,
+    funcID_morestack,
+    funcID_mstart,
+    funcID_rt0_go,
+    funcID_asmcgocall,
+    funcID_sigpanic,
+    funcID_runfinq,
+    funcID_gcBgMarkWorker,
+    funcID_systemstack_switch,
+    funcID_systemstack,
+    funcID_cgocallback_gofunc,
+    funcID_gogo,
+    funcID_externalthreadhandler,
+    funcID_debugCallV1,
+    funcID_gopanic,
+    funcID_panicwrap,
+    funcID_handleAsyncEvent,
+    funcID_asyncPreempt,
+    funcID_wrapper
+  };
+
+#define FuncFields                                                             \
+  F(uint64_t, true, Entry)                                                     \
+  F(int32_t, false, Name)                                                      \
+  F(uint32_t, false, Args)                                                     \
+  F(uint32_t, false, Deferreturn)                                              \
+  F(int32_t, false, Pcsp)                                                      \
+  F(int32_t, false, Pcfile)                                                    \
+  F(int32_t, false, Pcln)                                                      \
+  F(uint32_t, false, Npcdata)                                                  \
+  F(uint8_t, false, FuncID)                                                    \
+  F(uint8_t, false, Unused1)                                                   \
+  F(uint8_t, false, Unused2)                                                   \
+  F(uint8_t, false, Nfuncdata)
+
+  struct _Func {
+#define F(Type, IsPointer, Field) Type Field;
+    FuncFields
+#undef F
+  } __GoFunc;
+
+  uint32_t Pcdata[_PCDATA_MAX] = {};
+  uint64_t Funcdata[_FUNCDATA_MAX] = {};
+
+  int32_t PcdataMax[_PCDATA_MAX] = {};
+
+  void __read(const BinaryContext &BC, DataExtractor &DE,
+              BinarySection *Section, uint64_t *FuncOffset) override {
+#define F(Type, IsPointer, Field)                                              \
+  {                                                                            \
+    assert(DE.isValidOffset(*FuncOffset) && "Invalid offset");                 \
+    size_t FSize =                                                             \
+        IsPointer ? BC.AsmInfo->getCodePointerSize() : sizeof(Type);           \
+    if (IsPointer && Section)                                                  \
+      RemoveRelaReloc(BC, Section, *FuncOffset);                               \
+    __GoFunc.Field = readEndianVal(DE, FuncOffset, FSize);                     \
+  }
+    FuncFields
+#undef F
+  }
+
+  void __write(BinaryFunction *BF, uint8_t **FuncPart, uint8_t *SectionData,
+               BinarySection *OutputSection) const override {
+    BinaryContext &BC = BF->getBinaryContext();
+#define F(Type, IsPointer, Field)                                              \
+  {                                                                            \
+    if (IsPointer) {                                                           \
+      uint64_t Delta = (uint64_t)(*FuncPart - SectionData);                    \
+      AddRelaReloc(BC, BF->getSymbol(), OutputSection, Delta, 0);              \
+      *FuncPart += BC.AsmInfo->getCodePointerSize();                           \
+    } else {                                                                   \
+      writeEndianVal(BC, __GoFunc.Field, sizeof(Type), FuncPart);              \
+    }                                                                          \
+  }
+    FuncFields
+#undef F
+  }
+
+  size_t getSize(BinaryContext &BC) const override {
+    size_t FuncSize = 0;
+#define F(Type, IsPointerSize, Field)                                          \
+  if (IsPointerSize)                                                           \
+    FuncSize += BC.AsmInfo->getCodePointerSize();                              \
+  else                                                                         \
+    FuncSize += sizeof(Type);
+    FuncFields
+#undef F
+        return FuncSize;
+  }
+
+#undef FuncFields
+
+  void disableMetadata() override {
+    __GoFunc.Pcfile = 0;
+    __GoFunc.Pcln = 0;
+  }
+
+  int32_t getNameOffset() const override { return __GoFunc.Name; }
+
+  void setNameOffset(int32_t Offset) override { __GoFunc.Name = Offset; }
+
+  uint32_t getDeferreturnOffset() const override {
+    return __GoFunc.Deferreturn;
+  }
+
+  void setDeferreturnOffset(uint32_t Offset) override {
+    __GoFunc.Deferreturn = Offset;
+  }
+
+  uint32_t getPcspOffset() const override { return __GoFunc.Pcsp; }
+
+  void setPcspOffset(uint32_t Offset) override { __GoFunc.Pcsp = Offset; }
+
+  uint32_t getNpcdata() const override { return __GoFunc.Npcdata; }
+
+  void fixNpcdata() override {
+    for (int I = _PCDATA_MAX - 1; I >= 0; --I) {
+      if (Pcdata[I]) {
+        __GoFunc.Npcdata = I + 1;
+        return;
+      }
+    }
+
+    __GoFunc.Npcdata = 0;
+  }
+
+  bool hasReservedID(std::string Name) const override {
+    return __GoFunc.FuncID != funcID_normal &&
+           __GoFunc.FuncID != funcID_wrapper;
+  }
+
+  uint8_t getNfuncdata() const override { return __GoFunc.Nfuncdata; }
+
+  unsigned getPcdataUnsafePointIndex() const override {
+    return _PCDATA_RegMapIndex;
+  }
+
+  unsigned getPcdataStackMapIndex() const override {
+    return _PCDATA_StackMapIndex;
+  }
+
+  unsigned getPcdataInlTreeIndex() const override {
+    return _PCDATA_InlTreeIndex;
+  }
+
+  unsigned getPcdataMaxIndex() const override { return _PCDATA_MAX; }
+
+  size_t getPcdataSize() const override { return sizeof(Pcdata); }
+
+  uint32_t getPcdata(unsigned Index) const override {
+    assert(Index < _PCDATA_MAX && "Invalid index");
+    return Pcdata[Index];
+  }
+
+  void setPcdata(unsigned Index, uint32_t Value) override {
+    assert(Index < _PCDATA_MAX && "Invalid index");
+    Pcdata[Index] = Value;
+  }
+
+  void setPcdataMaxVal(unsigned Index, int32_t Value) override {
+    assert(Index < _PCDATA_MAX && "Invalid index");
+    PcdataMax[Index] = Value;
+  }
+
+  int32_t getPcdataMax(unsigned Index) const override {
+    assert(Index < _PCDATA_MAX && "Invalid index");
+    return PcdataMax[Index];
+  }
+
+  int getPcdataSafePointVal() const override {
+    const int Val = -1;
+    return Val;
+  }
+
+  int getPcdataUnsafePointVal() const override {
+    const int Val = -2;
+    return Val;
+  }
+
+  unsigned getFuncdataInlTreeIndex() const override {
+    return _FUNCDATA_InlTree;
+  }
+
+  uint64_t getFuncdata(unsigned Index) const override {
+    assert(Index < _FUNCDATA_MAX && "Invalid index");
+    return Funcdata[Index];
+  }
+
+  void setFuncdata(unsigned Index, uint64_t Value) override {
+    assert(Index < _FUNCDATA_MAX && "Invalid index");
+    Funcdata[Index] = Value;
+  }
+};
+
+struct Module_v1_14_9 : Module {
+  ~Module_v1_14_9() = default;
+
+  union ModuleStruct {
+    struct {
+      GoArray pclntable;
+
+      GoArray ftab;
+
+      GoArray filetab;
+
+      uint64_t findfunctab;
+      uint64_t minpc, maxpc;
+      uint64_t text, etext;
+      uint64_t noptrdata, enoptrdata;
+      uint64_t data, edata;
+      uint64_t bss, ebss;
+      uint64_t noptrbss, enoptrbss;
+      uint64_t end, gcdata, gcbss;
+      uint64_t types, etypes;
+
+      GoArray textsectmap;
+
+      GoArray typelinks;
+
+      GoArray itablinks;
+
+      // Other fields are zeroed/unused in exec
+    } m;
+
+    uint64_t a[sizeof(m) / sizeof(uint64_t)];
+  } ModuleStruct;
+
+  uint64_t getFieldOffset(BinaryContext &BC, uint64_t *Addr) const override {
+    unsigned Psize = BC.AsmInfo->getCodePointerSize();
+    return (Addr - ModuleStruct.a) * Psize;
+  }
+
+  uint64_t *getModule() override { return ModuleStruct.a; }
+
+  size_t getModuleSize() const override { return sizeof(ModuleStruct.m); }
+
+  void setPclntabSize(uint64_t Size) override {
+    // Set pclntable size
+    ModuleStruct.m.pclntable.setCount(Size);
+  }
+
+  void setFtabSize(uint64_t Count) override {
+    // Fix ftab size; the last entry reserved for maxpc
+    ModuleStruct.m.ftab.setCount(Count + 1);
+  }
+
+  uint64_t getPcHeaderAddr() const override {
+    return ModuleStruct.m.pclntable.getAddress();
+  }
+
+  const GoArray &getFtab() const override { return ModuleStruct.m.ftab; }
+
+  uint64_t getFindfunctab() const override {
+    return ModuleStruct.m.findfunctab;
+  }
+
+  uint64_t getTypes() const override { return ModuleStruct.m.types; }
+
+  uint64_t getEtypes() const override { return ModuleStruct.m.etypes; }
+
+  const GoArray &getTextsectmap() const override {
+    return ModuleStruct.m.textsectmap;
+  }
+
+  const GoArray &getTypelinks() const override {
+    return ModuleStruct.m.typelinks;
+  }
+
+  int patch(BinaryContext &BC) override {
+    BinaryData *Module = getModuleBD(BC);
+    if (!Module) {
+      errs() << "BOLT-ERROR: Failed to get firstmoduledata symbol!\n";
+      return -1;
+    }
+
+    BinarySection *Section = &Module->getSection();
+    std::vector<BinaryFunction *> BFs = BC.getSortedFunctions();
+    unsigned Psize = BC.AsmInfo->getCodePointerSize();
+
+#define getOffset(Field)                                                       \
+  Module->getOffset() + getFieldOffset(BC, &ModuleStruct.m.Field);
+
+#define getValue(Field) (ModuleStruct.m.Field)
+
+    // Fix firsmoduledata pointers
+    BinaryData *PclntabSym = BC.getBinaryDataAtAddress(getPcHeaderAddr());
+    assert(PclntabSym && "PclntabSym absent");
+    BinaryData *FindfunctabSym = BC.getBinaryDataAtAddress(getFindfunctab());
+    assert(FindfunctabSym && "FindfunctabSym absent");
+    BinaryFunction *FirstBF = getFirstBF(BC, BFs);
+    assert(FirstBF && "Text BF absent");
+    BinaryFunction *LastBF = getLastBF(BC, BFs);
+    assert(LastBF && "Text BF absent");
+
+#define FirstmoduleFields                                                      \
+  F(pclntable.Address, PclntabSym, 0)                                          \
+  F(ftab.Address, PclntabSym, Pclntab_v1_14_9::getPcHeaderSize(Psize))         \
+  F(findfunctab, FindfunctabSym, 0)                                            \
+  F(minpc, FirstBF, 0)                                                         \
+  F(text, FirstBF, 0)                                                          \
+  F(maxpc, LastBF, 0)                                                          \
+  F(etext, LastBF, 0)
+
+#define F(Field, Symbol, Addend)                                               \
+  {                                                                            \
+    uint64_t FieldOffset = getOffset(Field);                                   \
+    RemoveRelaReloc(BC, Section, FieldOffset);                                 \
+    AddRelaReloc(BC, Symbol->getSymbol(), Section, FieldOffset, Addend);       \
+  }
+    FirstmoduleFields
+#undef F
+#undef FirstmoduleFields
+
+        // Fix firstmoduledata static fields
+        MCSymbol *ZeroSym = BC.registerNameAtAddress("Zero", 0, 0, 0);
+#define FirstmoduleFields                                                      \
+  F(pclntable.Count[0])                                                        \
+  F(pclntable.Count[1])                                                        \
+  F(ftab.Count[0])                                                             \
+  F(ftab.Count[1])                                                             \
+  F(filetab.Count[0])                                                          \
+  F(filetab.Count[1])
+
+#define F(Field)                                                               \
+  {                                                                            \
+    uint64_t FieldOffset = getOffset(Field);                                   \
+    uint64_t FieldVal = getValue(Field);                                       \
+    Section->addRelocation(FieldOffset, ZeroSym, Relocation::getAbs(Psize),    \
+                           FieldVal);                                          \
+  }
+    FirstmoduleFields
+#undef F
+#undef FirstmoduleFields
+
+#undef getValue
+#undef getOffset
+        return 0;
+  }
+};
+
+class Pclntab_v1_14_12 : public Pclntab_v1_14_9 {
+public:
+  ~Pclntab_v1_14_12() = default;
+};
+
+struct GoFunc_v1_14_12 : GoFunc_v1_14_9 {
+  ~GoFunc_v1_14_12() = default;
+};
+
+struct Module_v1_14_12 : Module_v1_14_9 {
+  ~Module_v1_14_12() = default;
+};
+
+} // namespace bolt
+} // namespace llvm
+
+#endif

bolt/include/bolt/Passes/Golang/go_v1_16_5.h

+//===--------- Passes/Golang/go_v1_16_6.h -----------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVM_BOLT_GOLANG_V1_16_5_H
+#define LLVM_TOOLS_LLVM_BOLT_GOLANG_V1_16_5_H
+
+#include "go_base.h"
+
+namespace llvm {
+namespace bolt {
+
+class Pclntab_v1_16_5 : public Pclntab {
+#define PcHeaderFields                                                         \
+  F(uint32_t, false, Magic)                                                    \
+  F(uint8_t, false, Zero1)                                                     \
+  F(uint8_t, false, Zero2)                                                     \
+  F(uint8_t, false, MinLC)                                                     \
+  F(uint8_t, false, PtrSize)                                                   \
+  F(uint64_t, true, Nfuncs)                                                    \
+  F(uint64_t, true, Nfiles)                                                    \
+  F(uint64_t, true, FuncnameOffset)                                            \
+  F(uint64_t, true, CuOffset)                                                  \
+  F(uint64_t, true, FiletabOffset)                                             \
+  F(uint64_t, true, PctabOffset)                                               \
+  F(uint64_t, true, PclnOffset)
+
+  struct PcHeader {
+#define F(Type, IsPointerSize, Field) Type Field;
+    PcHeaderFields
+#undef F
+  } Header;
+
+  void __readHeader(BinaryContext &BC, DataExtractor &DE) override {
+    uint64_t Offset = getPclntabHeaderOffset();
+#define F(Type, IsPointer, Field)                                              \
+  {                                                                            \
+    assert(DE.isValidOffset(Offset) && "Invalid offset");                      \
+    size_t FSize =                                                             \
+        IsPointer ? BC.AsmInfo->getCodePointerSize() : sizeof(Type);           \
+    Header.Field = readEndianVal(DE, &Offset, FSize);                          \
+  }
+    PcHeaderFields
+#undef F
+  }
+
+  void __writeHeader(BinaryContext &BC, uint8_t *Pclntab) const override {
+#define F(Type, IsPointer, Field)                                              \
+  {                                                                            \
+    size_t FSize =                                                             \
+        IsPointer ? BC.AsmInfo->getCodePointerSize() : sizeof(Type);           \
+    writeEndianVal(BC, Header.Field, FSize, &Pclntab);                         \
+  }
+    PcHeaderFields
+#undef F
+  }
+
+  void setNewHeaderOffsets() override {
+    Header.FuncnameOffset = 0;
+    Header.CuOffset = 0;
+    Header.FiletabOffset = 0;
+    Header.PctabOffset = 0;
+    Header.PclnOffset = getPcHeaderSize();
+  }
+
+  bool checkMagic() const override { return Header.Magic == 0xfffffffa; }
+
+public:
+  ~Pclntab_v1_16_5() = default;
+
+  static size_t getPcHeaderSize(unsigned Psize) {
+    size_t FuncSize = 0;
+#define F(Type, IsPointerSize, Field)                                          \
+  if (IsPointerSize)                                                           \
+    FuncSize += Psize;                                                         \
+  else                                                                         \
+    FuncSize += sizeof(Type);
+    PcHeaderFields
+#undef F
+
+        return alignTo(FuncSize, Psize);
+  }
+
+#undef PcHeaderFields
+
+  size_t getPcHeaderSize() const override {
+    return getPcHeaderSize(Header.PtrSize);
+  }
+
+  void setFunctionsCount(uint64_t Count) override { Header.Nfuncs = Count; }
+
+  uint8_t getQuantum() const override { return Header.MinLC; }
+
+  uint8_t getPsize() const override { return Header.PtrSize; }
+
+  uint64_t getFunctionsCount() const override { return Header.Nfuncs; }
+
+  uint64_t getNameOffset() const override {
+    return getPclntabHeaderOffset() + Header.FuncnameOffset;
+  }
+
+  uint64_t getFiletabOffset() const override {
+    return getPclntabHeaderOffset() + Header.FiletabOffset;
+  }
+
+  uint64_t getPctabOffset() const override {
+    return getPclntabHeaderOffset() + Header.PctabOffset;
+  }
+
+  uint64_t getPclntabOffset() const override {
+    return getPclntabHeaderOffset() + Header.PclnOffset;
+  }
+
+  uint64_t getFunctabOffset() const override {
+    return getPclntabHeaderOffset() + Header.PclnOffset;
+  }
+};
+
+struct GoFunc_v1_16_5 : GoFunc {
+  ~GoFunc_v1_16_5() = default;
+
+  // runtime/symtab.go
+  enum {
+    _PCDATA_UnsafePoint = 0,
+    _PCDATA_StackMapIndex = 1,
+    _PCDATA_InlTreeIndex = 2,
+    _PCDATA_MAX,
+    _FUNCDATA_ArgsPointerMaps = 0,
+    _FUNCDATA_LocalsPointerMaps = 1,
+    _FUNCDATA_StackObjects = 2,
+    _FUNCDATA_InlTree = 3,
+    _FUNCDATA_OpenCodedDeferInfo = 4,
+    _FUNCDATA_ArgInfo = 5, // NOTE gc 1.17
+    _FUNCDATA_MAX,
+    _ArgsSizeUnknown = -0x80000000
+  };
+
+  // runtime/symtab.go
+  enum {
+    _PCDATA_UnsafePointSafe = -1,
+    _PCDATA_UnsafePointUnsafe = -2,
+    _PCDATA_Restart1 = -3,
+    _PCDATA_Restart2 = -4,
+    _PCDATA_RestartAtEntry = -5
+  };
+
+  // runtime/symtab.go
+  enum {
+    funcID_normal = 0,
+    funcID_runtime_main,
+    funcID_goexit,
+    funcID_jmpdefer,
+    funcID_mcall,
+    funcID_morestack,
+    funcID_mstart,
+    funcID_rt0_go,
+    funcID_asmcgocall,
+    funcID_sigpanic,
+    funcID_runfinq,
+    funcID_gcBgMarkWorker,
+    funcID_systemstack_switch,
+    funcID_systemstack,
+    funcID_cgocallback,
+    funcID_gogo,
+    funcID_externalthreadhandler,
+    funcID_debugCallV1,
+    funcID_gopanic,
+    funcID_panicwrap,
+    funcID_handleAsyncEvent,
+    funcID_asyncPreempt,
+    funcID_wrapper
+  };
+
+#define FuncFields                                                             \
+  F(uint64_t, true, Entry)                                                     \
+  F(int32_t, false, Name)                                                      \
+  F(int32_t, false, Args)                                                      \
+  F(uint32_t, false, Deferreturn)                                              \
+  F(uint32_t, false, Pcsp)                                                     \
+  F(uint32_t, false, pcfile)                                                   \
+  F(uint32_t, false, Pcln)                                                     \
+  F(uint32_t, false, Npcdata)                                                  \
+  F(uint32_t, false, CuOffset)                                                 \
+  F(uint8_t, false, FuncID)                                                    \
+  F(uint8_t, false, Flag)                                                      \
+  F(uint8_t, false, Unused2)                                                   \
+  F(uint8_t, false, Nfuncdata)
+
+  struct _Func {
+#define F(Type, IsPointer, Field) Type Field;
+    FuncFields
+#undef F
+  } __GoFunc;
+
+  uint32_t Pcdata[_PCDATA_MAX] = {};
+  uint64_t Funcdata[_FUNCDATA_MAX] = {};
+
+  int32_t PcdataMax[_PCDATA_MAX] = {};
+
+  void __read(const BinaryContext &BC, DataExtractor &DE,
+              BinarySection *Section, uint64_t *FuncOffset) override {
+#define F(Type, IsPointer, Field)                                              \
+  {                                                                            \
+    assert(DE.isValidOffset(*FuncOffset) && "Invalid offset");                 \
+    size_t FSize =                                                             \
+        IsPointer ? BC.AsmInfo->getCodePointerSize() : sizeof(Type);           \
+    if (IsPointer && Section)                                                  \
+      RemoveRelaReloc(BC, Section, *FuncOffset);                               \
+    __GoFunc.Field = readEndianVal(DE, FuncOffset, FSize);                     \
+  }
+    FuncFields
+#undef F
+  }
+
+  void __write(BinaryFunction *BF, uint8_t **FuncPart, uint8_t *SectionData,
+               BinarySection *OutputSection) const override {
+    BinaryContext &BC = BF->getBinaryContext();
+#define F(Type, IsPointer, Field)                                              \
+  {                                                                            \
+    if (IsPointer) {                                                           \
+      uint64_t Delta = (uint64_t)(*FuncPart - SectionData);                    \
+      AddRelaReloc(BC, BF->getSymbol(), OutputSection, Delta, 0);              \
+      *FuncPart += BC.AsmInfo->getCodePointerSize();                           \
+    } else {                                                                   \
+      writeEndianVal(BC, __GoFunc.Field, sizeof(Type), FuncPart);              \
+    }                                                                          \
+  }
+    FuncFields
+#undef F
+  }
+
+  size_t getSize(BinaryContext &BC) const override {
+    size_t FuncSize = 0;
+#define F(Type, IsPointerSize, Field)                                          \
+  if (IsPointerSize)                                                           \
+    FuncSize += BC.AsmInfo->getCodePointerSize();                              \
+  else                                                                         \
+    FuncSize += sizeof(Type);
+    FuncFields
+#undef F
+        return FuncSize;
+  }
+
+#undef FuncFields
+
+  void disableMetadata() override {
+    __GoFunc.pcfile = 0;
+    __GoFunc.Pcln = 0;
+    __GoFunc.CuOffset = 0;
+  }
+
+  int32_t getNameOffset() const override { return __GoFunc.Name; }
+
+  void setNameOffset(int32_t Offset) override { __GoFunc.Name = Offset; }
+
+  uint32_t getDeferreturnOffset() const override {
+    return __GoFunc.Deferreturn;
+  }
+
+  void setDeferreturnOffset(uint32_t Offset) override {
+    __GoFunc.Deferreturn = Offset;
+  }
+
+  uint32_t getPcspOffset() const override { return __GoFunc.Pcsp; }
+
+  void setPcspOffset(uint32_t Offset) override { __GoFunc.Pcsp = Offset; }
+
+  uint32_t getNpcdata() const override { return __GoFunc.Npcdata; }
+
+  void fixNpcdata() override {
+    for (int i = _PCDATA_MAX - 1; i >= 0; --i) {
+      if (Pcdata[i]) {
+        __GoFunc.Npcdata = i + 1;
+        return;
+      }
+    }
+
+    __GoFunc.Npcdata = 0;
+  }
+
+  bool hasReservedID(std::string Name) const override {
+    return __GoFunc.FuncID != funcID_normal &&
+           __GoFunc.FuncID != funcID_wrapper;
+  }
+
+  uint8_t getNfuncdata() const override { return __GoFunc.Nfuncdata; }
+
+  unsigned getPcdataUnsafePointIndex() const override {
+    return _PCDATA_UnsafePoint;
+  }
+
+  unsigned getPcdataStackMapIndex() const override {
+    return _PCDATA_StackMapIndex;
+  }
+
+  unsigned getPcdataInlTreeIndex() const override {
+    return _PCDATA_InlTreeIndex;
+  }
+
+  unsigned getPcdataMaxIndex() const override { return _PCDATA_MAX; }
+
+  size_t getPcdataSize() const override { return sizeof(Pcdata); }
+
+  uint32_t getPcdata(unsigned Index) const override {
+    assert(Index < _PCDATA_MAX && "Invalid index");
+    return Pcdata[Index];
+  }
+
+  void setPcdata(unsigned Index, uint32_t Value) override {
+    assert(Index < _PCDATA_MAX && "Invalid index");
+    Pcdata[Index] = Value;
+  }
+
+  void setPcdataMaxVal(unsigned Index, int32_t Value) override {
+    assert(Index < _PCDATA_MAX && "Invalid index");
+    PcdataMax[Index] = Value;
+  }
+
+  int32_t getPcdataMax(unsigned Index) const override {
+    return PcdataMax[Index];
+  }
+
+  int getPcdataSafePointVal() const override { return _PCDATA_UnsafePointSafe; }
+
+  int getPcdataUnsafePointVal() const override {
+    return _PCDATA_UnsafePointUnsafe;
+  }
+
+  unsigned getFuncdataInlTreeIndex() const override {
+    return _FUNCDATA_InlTree;
+  }
+
+  uint64_t getFuncdata(unsigned Index) const override {
+    assert(Index < _FUNCDATA_MAX && "Invalid index");
+    return Funcdata[Index];
+  }
+
+  void setFuncdata(unsigned Index, uint64_t Value) override {
+    assert(Index < _FUNCDATA_MAX && "Invalid index");
+    Funcdata[Index] = Value;
+  }
+};
+
+struct Module_v1_16_5 : Module {
+  ~Module_v1_16_5() = default;
+
+  union ModuleStruct {
+    struct {
+      uint64_t pcHeader;
+
+      // Function names part
+      GoArray funcnametab;
+
+      // Compilation Unit indexes part
+      GoArray cutab;
+
+      // Source file names part
+      GoArray filetab;
+
+      // Functions pc-relative part
+      GoArray pctab;
+
+      // Function - ftab offset table part
+      GoArray pclntable;
+
+      // Functions table part
+      GoArray ftab;
+
+      uint64_t findfunctab;
+      uint64_t minpc, maxpc;
+      uint64_t text, etext;
+      uint64_t noptrdata, enoptrdata;
+      uint64_t data, edata;
+      uint64_t bss, ebss;
+      uint64_t noptrbss, enoptrbss;
+      uint64_t end, gcdata, gcbss;
+      uint64_t types, etypes;
+
+      GoArray textsectmap;
+
+      GoArray typelinks;
+
+      GoArray itablinks;
+
+      // Other fields are zeroed/unused in exec
+    } m;
+
+    uint64_t a[sizeof(m) / sizeof(uint64_t)];
+  } ModuleStruct;
+
+  uint64_t getFieldOffset(BinaryContext &BC, uint64_t *Addr) const override {
+    unsigned Psize = BC.AsmInfo->getCodePointerSize();
+    return (Addr - ModuleStruct.a) * Psize;
+  }
+
+  uint64_t *getModule() override { return ModuleStruct.a; }
+
+  size_t getModuleSize() const override { return sizeof(ModuleStruct.m); }
+
+  void setPclntabSize(uint64_t Size) override {
+    // Set funcnametab size
+    ModuleStruct.m.funcnametab.setCount(Size);
+
+    // Set pctab size
+    ModuleStruct.m.pctab.setCount(Size);
+
+    // Set pclntable size
+    ModuleStruct.m.pclntable.setCount(Size);
+  }
+
+  void setFtabSize(uint64_t Count) override {
+    // Fix ftab size; the last entry reserved for maxpc
+    ModuleStruct.m.ftab.setCount(Count + 1);
+  }
+
+  uint64_t getPcHeaderAddr() const override { return ModuleStruct.m.pcHeader; }
+
+  const GoArray &getFtab() const override { return ModuleStruct.m.ftab; }
+
+  uint64_t getFindfunctab() const override {
+    return ModuleStruct.m.findfunctab;
+  }
+
+  uint64_t getTypes() const override { return ModuleStruct.m.types; }
+
+  uint64_t getEtypes() const override { return ModuleStruct.m.etypes; }
+
+  const GoArray &getTextsectmap() const override {
+    return ModuleStruct.m.textsectmap;
+  }
+
+  const GoArray &getTypelinks() const override {
+    return ModuleStruct.m.typelinks;
+  }
+
+  int patch(BinaryContext &BC) override {
+    BinaryData *Module = getModuleBD(BC);
+    if (!Module) {
+      errs() << "BOLT-ERROR: Failed to get firstmoduledata symbol!\n";
+      return -1;
+    }
+
+    BinarySection *Section = &Module->getSection();
+    std::vector<BinaryFunction *> BFs = BC.getSortedFunctions();
+    unsigned Psize = BC.AsmInfo->getCodePointerSize();
+
+#define getOffset(Field)                                                       \
+  Module->getOffset() + getFieldOffset(BC, &ModuleStruct.m.Field);
+
+#define getValue(Field) (ModuleStruct.m.Field)
+
+    // Fix firsmoduledata pointers
+    BinaryData *PclntabSym = BC.getBinaryDataAtAddress(getPcHeaderAddr());
+    assert(PclntabSym && "PclntabSym absent");
+    BinaryData *FindfunctabSym = BC.getBinaryDataAtAddress(getFindfunctab());
+    assert(FindfunctabSym && "FindfunctabSym absent");
+    BinaryFunction *FirstBF = getFirstBF(BC, BFs);
+    assert(FirstBF && "Text BF absent");
+    BinaryFunction *LastBF = getLastBF(BC, BFs);
+    assert(LastBF && "Text BF absent");
+
+#define FirstmoduleFields                                                      \
+  F(pcHeader, PclntabSym, 0)                                                   \
+  F(funcnametab.Address, PclntabSym, 0)                                        \
+  F(pctab.Address, PclntabSym, 0)                                              \
+  F(pclntable.Address, PclntabSym, 0)                                          \
+  F(ftab.Address, PclntabSym, Pclntab_v1_16_5::getPcHeaderSize(Psize))         \
+  F(findfunctab, FindfunctabSym, 0)                                            \
+  F(minpc, FirstBF, 0)                                                         \
+  F(text, FirstBF, 0)                                                          \
+  F(maxpc, LastBF, 0)                                                          \
+  F(etext, LastBF, 0)
+
+#define F(Field, Symbol, Addend)                                               \
+  {                                                                            \
+    uint64_t FieldOffset = getOffset(Field);                                   \
+    RemoveRelaReloc(BC, Section, FieldOffset);                                 \
+    AddRelaReloc(BC, Symbol->getSymbol(), Section, FieldOffset, Addend);       \
+  }
+    FirstmoduleFields
+#undef F
+#undef FirstmoduleFields
+
+        // Fix firstmoduledata static fields
+        MCSymbol *ZeroSym = BC.registerNameAtAddress("Zero", 0, 0, 0);
+#define FirstmoduleFields                                                      \
+  F(funcnametab.Count[0])                                                      \
+  F(funcnametab.Count[1])                                                      \
+  F(pctab.Count[0])                                                            \
+  F(pctab.Count[1])                                                            \
+  F(pclntable.Count[0])                                                        \
+  F(pclntable.Count[1])                                                        \
+  F(ftab.Count[0])                                                             \
+  F(ftab.Count[1])
+
+#define F(Field)                                                               \
+  {                                                                            \
+    uint64_t FieldOffset = getOffset(Field);                                   \
+    uint64_t FieldVal = getValue(Field);                                       \
+    Section->addRelocation(FieldOffset, ZeroSym, Relocation::getAbs(Psize),    \
+                           FieldVal);                                          \
+  }
+    FirstmoduleFields
+#undef F
+#undef FirstmoduleFields
+
+#undef getValue
+#undef getOffset
+        return 0;
+  }
+};
+
+} // namespace bolt
+} // namespace llvm
+
+#endif

bolt/include/bolt/Passes/Golang/go_v1_17.h

+//===--------- Passes/Golang/go_v1_17.h -------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVM_BOLT_GOLANG_V1_17_H
+#define LLVM_TOOLS_LLVM_BOLT_GOLANG_V1_17_H
+
+#include "go_v1_16_5.h"
+
+namespace llvm {
+namespace bolt {
+
+class Pclntab_v1_17_2 : public Pclntab_v1_16_5 {
+public:
+  ~Pclntab_v1_17_2() = default;
+};
+
+struct GoFunc_v1_17_2 : GoFunc_v1_16_5 {
+  // runtime/symtab.go
+  enum {
+    funcID_normal = 0,
+    funcID_abort,
+    funcID_asmcgocall,
+    funcID_asyncPreempt,
+    funcID_cgocallback,
+    funcID_debugCallV2,
+    funcID_gcBgMarkWorker,
+    funcID_goexit,
+    funcID_gogo,
+    funcID_gopanic,
+    funcID_handleAsyncEvent,
+    funcID_jmpdefer,
+    funcID_mcall,
+    funcID_morestack,
+    funcID_mstart,
+    funcID_panicwrap,
+    funcID_rt0_go,
+    funcID_runfinq,
+    funcID_runtime_main,
+    funcID_sigpanic,
+    funcID_systemstack,
+    funcID_systemstack_switch,
+    funcID_wrapper
+  };
+
+  bool hasReservedID(std::string Name) const override {
+    // NOTE Go 1.17 has a bug with function names containing '_' symbol.
+    // https://go-review.googlesource.com/c/go/+/396797
+    // The end of the name might contain .abi0 prefix
+    const char Rt0Go[] = "runtime.rt0_go";
+    const char StackSwitch[] = "runtime.systemstack_switch";
+
+    return (__GoFunc.FuncID != funcID_normal &&
+            __GoFunc.FuncID != funcID_wrapper) ||
+           !strncmp(Name.c_str(), Rt0Go, sizeof(Rt0Go) - 1) ||
+           !strncmp(Name.c_str(), StackSwitch, sizeof(StackSwitch) - 1);
+  }
+
+  ~GoFunc_v1_17_2() = default;
+};
+
+struct Module_v1_17_2 : Module_v1_16_5 {
+  ~Module_v1_17_2() = default;
+};
+
+class Pclntab_v1_17_5 : public Pclntab_v1_17_2 {
+public:
+  ~Pclntab_v1_17_5() = default;
+};
+
+struct GoFunc_v1_17_5 : GoFunc_v1_17_2 {
+  ~GoFunc_v1_17_5() = default;
+};
+
+struct Module_v1_17_5 : Module_v1_17_2 {
+  ~Module_v1_17_5() = default;
+};
+
+class Pclntab_v1_17_8 : public Pclntab_v1_17_5 {
+public:
+  ~Pclntab_v1_17_8() = default;
+};
+
+struct GoFunc_v1_17_8 : GoFunc_v1_17_5 {
+  ~GoFunc_v1_17_8() = default;
+};
+
+struct Module_v1_17_8 : Module_v1_17_5 {
+  ~Module_v1_17_8() = default;
+};
+
+} // namespace bolt
+} // namespace llvm
+
+#endif

bolt/include/bolt/Passes/IdenticalCodeFolding.h

 namespace llvm {
 namespace bolt {
 
 /// An optimization that replaces references to identical functions with
 /// references to a single one of them.
 ///
 class IdenticalCodeFolding : public BinaryFunctionPass {
 protected:
-  bool shouldOptimize(const BinaryFunction &BF) const override {
-    if (BF.hasUnknownControlFlow())
-      return false;
-    if (BF.isFolded())
-      return false;
-    if (BF.hasSDTMarker())
-      return false;
-    return BinaryFunctionPass::shouldOptimize(BF);
-  }
+  bool shouldOptimizeICF(const BinaryFunction &BF) const;
 
 public:
   explicit IdenticalCodeFolding(const cl::opt<bool> &PrintPass)
       : BinaryFunctionPass(PrintPass) {}
 
   const char *getName() const override { return "identical-code-folding"; }
   void runOnFunctions(BinaryContext &BC) override;
 };
 
 } // namespace bolt
 } // namespace llvm
 
 #endif

bolt/include/bolt/Passes/LongJmp.h

class LongJmpPass : public BinaryFunctionPass {
   /// Lookup closest stub local to \p Func.
   BinaryBasicBlock *lookupLocalStub(const BinaryBasicBlock &SourceBB,
                                     const MCInst &Inst, const MCSymbol *TgtSym,
                                     uint64_t DotAddress) const;
 
   /// Helper to identify whether \p Inst is branching to a stub
   bool usesStub(const BinaryFunction &Func, const MCInst &Inst) const;
 
+  /// Helper function to return delta offset between  \p DotAddress and target
+  uint64_t getTargetOffset(const BinaryContext &BC, uint64_t InstSize,
+                           uint64_t TargetAddress, uint64_t DotAddress) const;
+  uint64_t getTargetOffset(const BinaryFunction &Func, uint64_t InstSize,
+                           const MCSymbol *TgtSym, uint64_t DotAddress) const;
+  uint64_t getTargetOffset(const BinaryFunction &Func, const MCInst &Inst,
+                           uint64_t DotAddress) const;
+
   /// True if Inst is a branch that is out of range
   bool needsStub(const BinaryBasicBlock &BB, const MCInst &Inst,
                  uint64_t DotAddress) const;
 
+  /// Relax instruction \p II in \p BB in-place if necessary
+  bool relaxInstruction(BinaryBasicBlock &BB, BinaryBasicBlock::iterator &II,
+                        uint64_t DotAddress);
+
   /// Expand the range of the stub in StubBB if necessary
   bool relaxStub(BinaryBasicBlock &StubBB);
 
   /// Helper to resolve a symbol address according to our tentative layout
   uint64_t getSymbolAddress(const BinaryContext &BC, const MCSymbol *Target,
                             const BinaryBasicBlock *TgtBB) const;
 
+  enum RelaxRet {
+    NotModified = 0,  // No changes were made
+    InstrRelaxed = 1, // Instruction relaxations were made
+    StubsInserted = 2 // New stubs were inserted in function
+  };
+
   /// Relax function by adding necessary stubs or relaxing existing stubs
-  bool relax(BinaryFunction &BF);
+  RelaxRet relax(BinaryFunction &BF);
 
 public:
   /// BinaryPass public interface
 
   explicit LongJmpPass(const cl::opt<bool> &PrintPass)
       : BinaryFunctionPass(PrintPass) {}
 
   const char *getName() const override { return "long-jmp"; }
 
   void runOnFunctions(BinaryContext &BC) override;
 };
 } // namespace bolt
 } // namespace llvm
 
 #endif

bolt/include/bolt/Utils/CommandLineOpts.h

 
 extern llvm::cl::opt<ProfileFormatKind> ProfileFormat;
 extern llvm::cl::opt<bool> SplitEH;
 extern llvm::cl::opt<bool> StrictMode;
 extern llvm::cl::opt<bool> TimeOpts;
 extern llvm::cl::opt<bool> UseOldText;
 extern llvm::cl::opt<bool> UpdateDebugSections;
 
+enum GolangVersion : char {
+  GV_NONE = 0, /// Do not use golang passes
+  GV_FIRST,
+  GV_AUTO = GV_FIRST, /// Auto delect golang version
+  GV_1_14_9,          /// Version 1.14.9
+  GV_1_14_12,         /// Version 1.14.12
+  GV_1_16_5,          /// Version 1.16.5
+  GV_1_17_2,          /// Version 1.17.2
+  GV_1_17_5,          /// Version 1.17.5
+  GV_1_17_8,          /// Version 1.17.8
+  GV_LAST,
+  GV_LATEST = GV_LAST - 1 /// Go latest version
+};
+
+extern llvm::cl::opt<GolangVersion> GolangPass;
 // The default verbosity level (0) is pretty terse, level 1 is fairly
 // verbose and usually prints some informational message for every
 // function processed.  Level 2 is for the noisiest of messages and
 // often prints a message per basic block.
 // Error messages should never be suppressed by the verbosity level.
 // Only warnings and info messages should be affected.
 //
 // The rationale behind stream usage is as follows:

bolt/include/bolt/Utils/NameResolver.h

class NameResolver {
   StringMap<uint64_t> Counters;
 
   /// Character guaranteed not to be used by any "native" name passed to
   /// uniquify() function.
   static constexpr char Sep = '/';
 
 public:
   /// Return unique version of the \p Name in the form "Name<Sep><Number>".
+  static std::string uniquifyID(StringRef Name, uint64_t ID) {
+    return (Name + Twine(Sep) + Twine(ID)).str();
+  }
+
   std::string uniquify(StringRef Name) {
     const uint64_t ID = ++Counters[Name];
-    return (Name + Twine(Sep) + Twine(ID)).str();
+    return uniquifyID(Name, ID);
   }
 
   /// For uniquified \p Name, return the original form (that may no longer be
   /// unique).
   static StringRef restore(StringRef Name) {
     return Name.substr(0, Name.find_first_of(Sep));
   }
 

bolt/lib/Core/BinaryBasicBlock.cpp

for (const auto &LocSymKV : *LocSyms) {
     if (getFunction()->requiresAddressTranslation()) {
       getFunction()->getInputOffsetToAddressMap().emplace(
           std::make_pair(InputFunctionOffset, OutputOffset + BBAddress));
     }
   }
   LocSyms.reset(nullptr);
 }
 
+bool BinaryBasicBlock::storeInstructionMetadata(BinaryBasicBlock::iterator II) {
+  // In case the instruction has locked annotation we will save all
+  // its annotations to noop instruction and store it in place of
+  // erased instruction
+  BinaryContext &BC = Function->getBinaryContext();
+  if (!BC.MIB->hasAnnotation(*II, "Locked"))
+    return false;
+
+  MCInst Noop;
+  BC.MIB->createNoop(Noop);
+  std::unique_lock<llvm::sys::RWMutex> Lock(BC.CtxMutex);
+  BC.MIB->copyAnnotationInst(*II, Noop);
+  Instructions[std::distance(Instructions.begin(), II)] = Noop;
+  return true;
+}
+
 } // namespace bolt
 } // namespace llvm

bolt/lib/Core/BinaryContext.cpp

while (BFI != BinaryFunctions.end()) {
       llvm::move(FunctionRanges, std::back_inserter(OutputRanges));
       std::advance(BFI, 1);
     }
   }
 
   return OutputRanges;
 }
 
+std::pair<unsigned, unsigned>
+BinaryContext::getBFAlignment(BinaryFunction &Function,
+                              bool EmitColdPart) const {
+  unsigned Alignment = Function.getAlignment();
+  if (HasRelocations) {
+    unsigned MaxAlignBytes = EmitColdPart ? Function.getMaxColdAlignmentBytes()
+                                          : Function.getMaxAlignmentBytes();
+    return std::make_pair(Alignment, MaxAlignBytes);
+  }
+
+  return std::make_pair(Alignment, Alignment);
+}
+
 } // namespace bolt
 } // namespace llvm

bolt/lib/Core/BinaryEmitter.cpp

if ((opts::AlignBlocks || opts::PreserveBlocksAlignment) &&
                                  BB->getAlignmentMaxBytes());
     Streamer.emitLabel(BB->getLabel());
     if (!EmitCodeOnly) {
       if (MCSymbol *EntrySymbol = BF.getSecondaryEntryPointSymbol(*BB))
         Streamer.emitLabel(EntrySymbol);
     }
 
     // Check if special alignment for macro-fusion is needed.
-    bool MayNeedMacroFusionAlignment =
-        (opts::AlignMacroOpFusion == MFT_ALL) ||
-        (opts::AlignMacroOpFusion == MFT_HOT && BB->getKnownExecutionCount());
+    bool MayNeedMacroFusionAlignment = ((opts::AlignMacroOpFusion == MFT_ALL) ||
+                                        (opts::AlignMacroOpFusion == MFT_HOT &&
+                                         BB->getKnownExecutionCount())) &&
+                                       opts::GolangPass == opts::GV_NONE;
     BinaryBasicBlock::const_iterator MacroFusionPair;
     if (MayNeedMacroFusionAlignment) {
       MacroFusionPair = BB->getMacroOpFusionPair();
       if (MacroFusionPair == BB->end())
         MayNeedMacroFusionAlignment = false;
     }
 
     SMLoc LastLocSeen;
for (BinarySection &Section : BC.sections()) {
     if (!Section.hasRelocations())
       continue;
 
     StringRef Prefix = Section.hasSectionRef() ? OrgSecPrefix : "";
     Section.emitAsData(Streamer, Prefix + Section.getName());
     Section.clearRelocations();
   }
 }
 
 namespace llvm {
 namespace bolt {
 
 void emitBinaryContext(MCStreamer &Streamer, BinaryContext &BC,
                        StringRef OrgSecPrefix) {
   BinaryEmitter(Streamer, BC).emitAll(OrgSecPrefix);
 }
 
 void emitFunctionBody(MCStreamer &Streamer, BinaryFunction &BF,
                       FunctionFragment &FF, bool EmitCodeOnly) {
   BinaryEmitter(Streamer, BF.getBinaryContext())
       .emitFunctionBody(BF, FF, EmitCodeOnly);
 }
 
 } // namespace bolt
 } // namespace llvm

bolt/lib/Core/BinaryFunction.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "bolt/Core/BinaryFunction.h"
 #include "bolt/Core/BinaryBasicBlock.h"
 #include "bolt/Core/BinaryDomTree.h"
 #include "bolt/Core/DynoStats.h"
 #include "bolt/Core/MCPlusBuilder.h"
+#include "bolt/Passes/Golang.h"
 #include "bolt/Utils/NameResolver.h"
 #include "bolt/Utils/NameShortener.h"
 #include "bolt/Utils/Utils.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Demangle/Demangle.h"
for (BinaryBasicBlock *BB : BasicBlocks) {
     // Also mark BBs with indirect jumps as reachable, since we do not
     // support removing unused jump tables yet (GH-issue20).
     for (const MCInst &Inst : *BB) {
       if (BC.MIB->getJumpTable(Inst)) {
         Stack.push(BB);
         BB->markValid(true);
         break;
       }
+
+      // NOTE GO: Deferreturn calls are located in unreachable regions
+      if (opts::GolangPass != opts::GV_NONE && BC.MIB->isCall(Inst) &&
+          BC.MIB->hasAnnotation(Inst, "IsDefer")) {
+        Stack.push(BB);
+        BB->markValid(true);
+        break;
+      }
     }
   }
 
   // Determine reachable BBs from the entry point
   while (!Stack.empty()) {
     BinaryBasicBlock *BB = Stack.top();
     Stack.pop();
     for (BinaryBasicBlock *Succ : BB->successors()) {
for (uint64_t Offset = 0; Offset < getSize(); Offset += Size) {
     const uint64_t AbsoluteInstrAddr = getAddress() + Offset;
 
     // Check for data inside code and ignore it
     if (const size_t DataInCodeSize = getSizeOfDataInCodeAt(Offset)) {
       Size = DataInCodeSize;
       continue;
     }
 
-    if (!BC.SymbolicDisAsm->getInstruction(Instruction, Size,
-                                           FunctionData.slice(Offset),
-                                           AbsoluteInstrAddr, nulls())) {
-      // Functions with "soft" boundaries, e.g. coming from assembly source,
-      // can have 0-byte padding at the end.
-      if (isZeroPaddingAt(Offset))
-        break;
-
+    auto disasmFailed = [&]() {
       errs() << "BOLT-WARNING: unable to disassemble instruction at offset 0x"
              << Twine::utohexstr(Offset) << " (address 0x"
              << Twine::utohexstr(AbsoluteInstrAddr) << ") in function " << *this
              << '\n';
       // Some AVX-512 instructions could not be disassembled at all.
       if (BC.HasRelocations && opts::TrapOnAVX512 && BC.isX86()) {
         setTrapOnEntry();
         BC.TrappedFunctions.push_back(this);
       } else {
         setIgnored();
       }
+    };
+
+    if (!BC.SymbolicDisAsm->getInstruction(Instruction, Size,
+                                           FunctionData.slice(Offset),
+                                           AbsoluteInstrAddr, nulls())) {
+      if (opts::GolangPass != opts::GV_NONE && BC.isAArch64()) {
+        // Golang uses special UND instruction for aarch64, handle it as NOP
+        // Also skipPleaseUseCallersFrames is full of 0, replace them with NOPs
+        DataExtractor DE =
+            DataExtractor(FunctionData, BC.AsmInfo->isLittleEndian(),
+                          BC.AsmInfo->getCodePointerSize());
+        uint64_t ValOffset = Offset;
+        uint32_t Value = DE.getU32(&ValOffset);
+        if (Value && Value != GolangPass::getUndAarch64()) {
+          disasmFailed();
+          errs() << "BOLT-INFO: Check that the binary was compiled with "
+                 << "-mappingsymbol option\n";
+          exit(1);
+        }
 
+        BC.MIB->createNoop(Instruction);
+      } else if (isZeroPaddingAt(Offset)) {
+        // Functions with "soft" boundaries, e.g. coming from assembly source,
+        // can have 0-byte padding at the end.
+        break;
+      } else {
+        disasmFailed();
         break;
       }
+    }
 
     // Check integrity of LLVM assembler/disassembler.
     if (opts::CheckEncoding && !BC.MIB->isBranch(Instruction) &&
         !BC.MIB->isCall(Instruction) && !BC.MIB->isNoop(Instruction)) {
       if (!BC.validateInstructionEncoding(FunctionData.slice(Offset, Size))) {
         errs() << "BOLT-WARNING: mismatching LLVM encoding detected in "
                << "function " << *this << " for instruction :\n";
         BC.printInstruction(errs(), Instruction, AbsoluteInstrAddr);
bool IsSDTMarker =
         MIB->isNoop(Instr) && BC.SDTMarkers.count(InstrInputAddr);
     bool IsLKMarker = BC.LKMarkers.count(InstrInputAddr);
     // Mark all nops with Offset for profile tracking purposes.
     if (MIB->isNoop(Instr) || IsLKMarker) {
       if (!MIB->getOffset(Instr))
         MIB->setOffset(Instr, static_cast<uint32_t>(Offset), AllocatorId);
       if (IsSDTMarker || IsLKMarker)
         HasSDTMarker = true;
-      else
+      else if (!PreserveNops)
         // Annotate ordinary nops, so we can safely delete them if required.
         MIB->addAnnotation(Instr, "NOP", static_cast<uint32_t>(1), AllocatorId);
     }
 
     if (!InsertBB) {
       // It must be a fallthrough or unreachable code. Create a new block unless
       // we see an unconditional branch following a conditional one. The latter
       // should not be a conditional tail call.
void BinaryFunction::postProcessCFG() {
   calculateMacroOpFusionStats();
 
   // The final cleanup of intermediate structures.
   clearList(IgnoredBranches);
 
   // Remove "Offset" annotations, unless we need an address-translation table
   // later. This has no cost, since annotations are allocated by a bumpptr
   // allocator and won't be released anyway until late in the pipeline.
-  if (!requiresAddressTranslation() && !opts::Instrument) {
+  if (!requiresAddressTranslation() && !opts::Instrument &&
+      opts::GolangPass == opts::GV_NONE) {
     for (BinaryBasicBlock &BB : blocks())
       for (MCInst &Inst : BB)
         BC.MIB->clearOffset(Inst);
   }
 
   assert((!isSimple() || validateCFG()) &&
          "invalid CFG detected after post-processing");
 }

bolt/lib/Core/BinarySection.cpp

BinarySection::hash(const BinaryData &BD,
 
   Cache[&BD] = Hash;
 
   return Hash;
 }
 
 void BinarySection::emitAsData(MCStreamer &Streamer,
                                const Twine &SectionName) const {
-  StringRef SectionContents = getContents();
+  StringRef SectionContents = isChanged() ? getOutputContents() : getContents();
   MCSectionELF *ELFSection =
       BC.Ctx->getELFSection(SectionName, getELFType(), getELFFlags());
 
   Streamer.switchSection(ELFSection);
   Streamer.emitValueToAlignment(getAlign());
 
   if (BC.HasRelocations && opts::HotData && isReordered())
     Streamer.emitLabel(BC.Ctx->getOrCreateSymbol("__hot_data_start"));

bolt/lib/Core/Relocation.cpp

 }
 
 bool Relocation::isPCRelative(uint64_t Type) {
   if (Arch == Triple::aarch64)
     return isPCRelativeAArch64(Type);
   return isPCRelativeX86(Type);
 }
 
+uint64_t Relocation::getAbs32() {
+  if (Arch == Triple::aarch64)
+    return ELF::R_AARCH64_ABS32;
+  return ELF::R_X86_64_32;
+}
+
 uint64_t Relocation::getAbs64() {
   if (Arch == Triple::aarch64)
     return ELF::R_AARCH64_ABS64;
   return ELF::R_X86_64_64;
 }
 
+uint64_t Relocation::getAbs(uint8_t Size) {
+  if (Size == sizeof(uint64_t))
+    return Relocation::getAbs64();
+  else if (Size == sizeof(uint32_t))
+    return Relocation::getAbs32();
+
+  llvm_unreachable("Wrong relocation size");
+  return 0;
+}
+
 uint64_t Relocation::getRelative() {
   if (Arch == Triple::aarch64)
     return ELF::R_AARCH64_RELATIVE;
   return ELF::R_X86_64_RELATIVE;
 }
 
 size_t Relocation::emit(MCStreamer *Streamer) const {
   const size_t Size = getSizeForType(Type);

bolt/lib/Passes/BinaryPasses.cpp

 }
 
 static cl::opt<bool> MinBranchClusters(
     "min-branch-clusters",
     cl::desc("use a modified clustering algorithm geared towards minimizing "
              "branches"),
     cl::Hidden, cl::cat(BoltOptCategory));
 
-static cl::list<Peepholes::PeepholeOpts> Peepholes(
+cl::list<Peepholes::PeepholeOpts> Peepholes(
     "peepholes", cl::CommaSeparated, cl::desc("enable peephole optimizations"),
     cl::value_desc("opt1,opt2,opt3,..."),
     cl::values(clEnumValN(Peepholes::PEEP_NONE, "none", "disable peepholes"),
                clEnumValN(Peepholes::PEEP_DOUBLE_JUMPS, "double-jumps",
                           "remove double jumps when able"),
                clEnumValN(Peepholes::PEEP_TAILCALL_TRAPS, "tailcall-traps",
                           "insert tail call traps"),
                clEnumValN(Peepholes::PEEP_USELESS_BRANCHES, "useless-branches",
for (BinaryFunction *Function : BC.getInjectedBinaryFunctions()) {
     Function->setCodeSectionName(BC.getInjectedCodeSectionName());
     Function->setColdCodeSectionName(BC.getInjectedColdCodeSectionName());
   }
 
   // In non-relocation mode functions have pre-assigned section names.
   if (!BC.HasRelocations)
     return;
 
-  const bool UseColdSection =
-      BC.NumProfiledFuncs > 0 ||
+  bool UseColdSection = BC.NumProfiledFuncs > 0 ||
                         opts::ReorderFunctions == ReorderFunctions::RT_USER;
+
+  // With RT_NONE reordering we won't assign indexes for BF
+  if (opts::ReorderFunctions == ReorderFunctions::RT_NONE)
+    UseColdSection = false;
+
   for (auto &BFI : BC.getBinaryFunctions()) {
     BinaryFunction &Function = BFI.second;
     if (opts::isHotTextMover(Function)) {
       Function.setCodeSectionName(BC.getHotTextMoverSectionName());
       Function.setColdCodeSectionName(BC.getHotTextMoverSectionName());
       continue;
     }
 
     if (!UseColdSection || Function.hasValidIndex())
       Function.setCodeSectionName(BC.getMainCodeSectionName());
     else
       Function.setCodeSectionName(BC.getColdCodeSectionName());
 
     if (Function.isSplit())
       Function.setColdCodeSectionName(BC.getColdCodeSectionName());
   }
 }
 

bolt/lib/Passes/CMakeLists.txt

 add_llvm_library(LLVMBOLTPasses
   ADRRelaxationPass.cpp
   Aligner.cpp
   AllocCombiner.cpp
   AsmDump.cpp
   BinaryPasses.cpp
   BinaryFunctionCallGraph.cpp
   CMOVConversion.cpp
   CacheMetrics.cpp
   CallGraph.cpp
   CallGraphWalker.cpp
   DataflowAnalysis.cpp
   DataflowInfoManager.cpp
   FrameAnalysis.cpp
   FrameOptimizer.cpp
   FixRelaxationPass.cpp
+  Golang.cpp
+  Golang-postprocess.cpp
+  Golang-preprocess.cpp
   HFSort.cpp
   HFSortPlus.cpp
   Hugify.cpp
   IdenticalCodeFolding.cpp
   IndirectCallPromotion.cpp
   Inliner.cpp
   Instrumentation.cpp
   JTFootprintReduction.cpp

bolt/lib/Passes/Golang-postprocess.cpp

+//===--------- Passes/Golang-postprocess.h---------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+//===----------------------------------------------------------------------===//
+
+#include "bolt/Core/ParallelUtilities.h"
+#include "bolt/Passes/Golang.h"
+
+#define DEBUG_TYPE "bolt-golang-postprocess"
+
+using namespace llvm;
+
+namespace opts {
+extern cl::opt<bool> Instrument;
+extern cl::opt<bool> GolangPcspPreserve;
+} // end namespace opts
+
+namespace llvm {
+namespace bolt {
+
+void GolangPostPass::skipPleaseUseCallersFramesPass(BinaryContext &BC) {
+  // The function must have sizeofSkipFunction bytes of nops
+  // runtime/traceback.go
+  const int sizeofSkipFunction = 256;
+  const char *FunctionName = "runtime.skipPleaseUseCallersFrames";
+  BinaryFunction *Function = BC.getBinaryFunctionByName(FunctionName);
+  if (!Function) {
+    LLVM_DEBUG(outs() << "BOLT-WARNING: Failed to get " << FunctionName
+                      << "\n");
+    return;
+  }
+
+  assert(Function->getLayout().block_begin() !=
+             Function->getLayout().block_end() &&
+         "Unexpected empty function");
+  BinaryBasicBlock *BB = *Function->getLayout().block_begin();
+  BB->clear();
+
+  MCInst Inst;
+  BC.MIB->createNoop(Inst);
+  uint64_t Size = sizeofSkipFunction / BC.computeInstructionSize(Inst);
+  std::unique_ptr<struct GoFunc> GoFunc = createGoFunc();
+  unsigned PcspIndex = GoFunc->getPcspIndex();
+  while (Size--) {
+    Inst.clear();
+    BC.MIB->createNoop(Inst);
+    BB->insertInstruction(BB->begin(), Inst);
+    addVarintAnnotation(BC, *BB->begin(), PcspIndex, /* Value */ 0,
+                        /*IsNext*/ false);
+  }
+}
+
+void GolangPostPass::instrumentExitCall(BinaryContext &BC) {
+  // Golang doesn not call anything on process termination
+  // Insert instrumentation fini call at exit function
+  if (!opts::Instrument)
+    return;
+
+  BinaryFunction *Function = BC.getBinaryFunctionByName("runtime.exit");
+  if (!Function) {
+    outs() << "BOLT-WARNING: Failed to get runtime.exit for instrumentation!\n";
+    return;
+  }
+
+  if (!BC.shouldEmit(*Function)) {
+    outs() << "BOLT-WARNING: runtime.exit could not be patched for "
+              "instrumentation!\n";
+    return;
+  }
+
+  assert(Function->getLayout().block_begin() !=
+             Function->getLayout().block_end() &&
+         "runtime.exit is empty");
+  BinaryBasicBlock *BB = *Function->getLayout().block_begin();
+  MCSymbol *FiniHandler =
+      BC.Ctx->getOrCreateSymbol("__bolt_trampoline_instr_fini_call");
+  std::vector<MCInst> Instrs = BC.MIB->createInstrumentFiniCall(
+      FiniHandler, &*BC.Ctx, /*IsTailCall*/ false);
+  BB->insertInstructions(BB->begin(), Instrs);
+}
+
+uint32_t GolangPostPass::pcdataPass(BinaryFunction *BF, GoFunc *GoFunc,
+                                    const uint32_t Index,
+                                    const unsigned AllocId) {
+  int Ret;
+  int32_t Val, NextVal;
+  MCInst NoopInst;
+  BinaryContext &BC = BF->getBinaryContext();
+  for (auto BBIt = BF->getLayout().block_begin();
+       BBIt != BF->getLayout().block_end(); ++BBIt) {
+    BinaryBasicBlock *BB = *BBIt;
+    for (uint64_t I = 0; I < BB->size(); ++I) {
+      MCInst &Inst = BB->getInstructionAtIndex(I);
+      bool IsMap = hasVarintAnnotation(BC, Inst, Index);
+      if (!IsMap)
+        continue;
+
+      if (Index == GoFunc->getPcdataStackMapIndex()) {
+        auto addNoop = [&](const int32_t Val, const int32_t NextVal) {
+          BC.MIB->createNoop(NoopInst);
+          addVarintAnnotation(BC, NoopInst, Index, Val, /*IsNext*/ false,
+                              AllocId);
+          addVarintAnnotation(BC, NoopInst, Index, NextVal, /*IsNext*/ true,
+                              AllocId);
+          auto NextIt = std::next(BB->begin(), I + 1);
+          BB->insertInstruction(NextIt, NoopInst);
+        };
+
+        NextVal = getVarintAnnotation(BC, Inst, Index, /*IsNext*/ true);
+        int32_t NextInstVal;
+        Ret = getNextMCinstVal(BBIt, I, Index, NextInstVal, nullptr);
+        if (Ret < 0) {
+          Val = getVarintAnnotation(BC, Inst, Index);
+          // Check that the last instruction value equals next value
+          if (Val != NextVal)
+            addNoop(NextVal, NextVal);
+
+          return 0;
+        }
+
+        // We need to save original chain of values for call instructions so
+        // check that the value of the next instuction is the same as expected
+        if (NextVal == NextInstVal || !BC.MIB->isCall(Inst))
+          continue;
+
+        // If the Value is not the same as expected create nop
+        // with the right values
+        addNoop(NextVal, NextInstVal);
+      }
+    }
+  }
+
+  return 0;
+}
+
+int GolangPostPass::pclntabPass(BinaryContext &BC) {
+  const uint64_t PclntabAddr = getPcHeaderAddr();
+  if (!PclntabAddr) {
+    errs() << "BOLT-ERROR: Pclntab address is zero!\n";
+    return -1;
+  }
+
+  BinaryData *PclntabSym = BC.getBinaryDataAtAddress(PclntabAddr);
+  if (!PclntabSym) {
+    errs() << "BOLT-ERROR: Failed to get pclntab symbol!\n";
+    return -1;
+  }
+
+  BinarySection *Section = &PclntabSym->getSection();
+  DataExtractor DE = DataExtractor(Section->getContents(),
+                                   BC.AsmInfo->isLittleEndian(), getPsize());
+
+  ParallelUtilities::WorkFuncWithAllocTy WorkFun =
+      [&](BinaryFunction &Function, MCPlusBuilder::AllocatorIdTy AllocId) {
+        uint64_t FuncOffset = Function.getGolangFunctabOffset();
+        std::unique_ptr<struct GoFunc> GoFunc = createGoFunc();
+        GoFunc->read(BC, DE, nullptr, &FuncOffset);
+
+        auto getPcdata = [&](const uint32_t Index, bool Force = false) {
+          uint32_t MapOffsetVal = GoFunc->getPcdata(Index);
+          if (!MapOffsetVal && !Force)
+            return;
+
+          pcdataPass(&Function, GoFunc.get(), Index, AllocId);
+        };
+
+        getPcdata(GoFunc->getPcdataStackMapIndex());
+
+        if (Function.getLayout().block_begin() !=
+            Function.getLayout().block_end()) {
+          // Insert NOP to the end of function, if it ends with call instruction
+          // to provide correct PCSP table lately for runtime.gentraceback
+          // This is needed for rare cases with no-return calls, since the
+          // pcvalue searches for targetpc < pc and for tail cals we will have
+          // targetpc == pc. We could also add +1 in PCSP final offset, but
+          // it won't fix the preserve PCSP table case, so this solution seems
+          // to be more robust
+          BinaryBasicBlock *BBend = *Function.getLayout().block_rbegin();
+          if (BC.MIB->isCall(*BBend->rbegin())) {
+            MCInst NoopInst;
+            BC.MIB->createNoop(NoopInst);
+            BBend->insertInstruction(BBend->end(), NoopInst);
+          }
+        }
+      };
+  ParallelUtilities::PredicateTy skipFunc =
+      [&](const BinaryFunction &Function) { return !Function.isGolang(); };
+
+  ParallelUtilities::runOnEachFunctionWithUniqueAllocId(
+      BC, ParallelUtilities::SchedulingPolicy::SP_INST_QUADRATIC, WorkFun,
+      skipFunc, "pcdataGoPostProcess", /*ForceSequential*/ true);
+  return 0;
+}
+
+void GolangPostPass::runOnFunctions(BinaryContext &BC) {
+  int Ret;
+
+  skipPleaseUseCallersFramesPass(BC);
+
+  instrumentExitCall(BC);
+
+  Ret = pclntabPass(BC);
+  if (Ret < 0) {
+    errs() << "BOLT-ERROR: Golang postprocess pclntab pass failed!\n";
+    exit(1);
+  }
+}
+
+} // end namespace bolt
+} // end namespace llvm

bolt/lib/Passes/Golang-preprocess.cpp

+//===--------- Passes/Golang-preprocess.h----------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+//===----------------------------------------------------------------------===//
+
+#include "bolt/Core/ParallelUtilities.h"
+#include "bolt/Passes/Golang.h"
+
+#define DEBUG_TYPE "bolt-golang-preprocess"
+
+using namespace llvm;
+using namespace bolt;
+
+namespace opts {
+extern cl::opt<bool> GolangPcspPreserve;
+} // end namespace opts
+
+namespace llvm {
+namespace bolt {
+
+static void inlTreePass(BinaryFunction *Function, GoFunc *GoFunc,
+                        const unsigned AllocId, const MCCodeEmitter *Emitter) {
+  BinaryContext &BC = Function->getBinaryContext();
+  const unsigned PcdataIndex = GoFunc->getPcdataInlTreeIndex();
+  int32_t MaxVal = GoFunc->getPcdataMax(PcdataIndex);
+  if (MaxVal < 0)
+    return;
+
+  const unsigned Index = GoFunc->getFuncdataInlTreeIndex();
+  uint64_t FuncdataAddr = GoFunc->getFuncdata(Index);
+  if (!FuncdataAddr)
+    return;
+
+  struct InlinedCall InlinedCall;
+  ErrorOr<BinarySection &> FuncdataSection =
+      BC.getSectionForAddress(FuncdataAddr);
+  if (!FuncdataSection)
+    return;
+
+  DataExtractor DE = DataExtractor(FuncdataSection->getContents(),
+                                   BC.AsmInfo->isLittleEndian(),
+                                   BC.AsmInfo->getCodePointerSize());
+
+  for (uint32_t I = 0; I < (uint32_t)MaxVal + 1; ++I) {
+    uint64_t Offset = (uint64_t)(FuncdataAddr - FuncdataSection->getAddress());
+    Offset += I * sizeof(InlinedCall);
+    assert(DE.isValidOffset(Offset) && "Invalid offset");
+    InlinedCall.Parent = (int16_t)DE.getU16(&Offset);
+    InlinedCall.FuncID = DE.getU8(&Offset);
+    InlinedCall.Unused = DE.getU8(&Offset);
+    InlinedCall.File = (int32_t)DE.getU32(&Offset);
+    InlinedCall.Line = (int32_t)DE.getU32(&Offset);
+    InlinedCall.Func = (int32_t)DE.getU32(&Offset);
+    InlinedCall.ParentPc = (int32_t)DE.getU32(&Offset);
+
+    for (BinaryBasicBlock *BB : Function->getLayout().blocks()) {
+      if (InlinedCall.ParentPc >= BB->getEndOffset())
+        continue;
+
+      uint32_t Offset = BB->getOffset();
+      for (MCInst &II : *BB) {
+        if (BC.MIB->hasAnnotation(II, "Offset")) {
+          constexpr size_t InvalidOffset = std::numeric_limits<uint32_t>::max();
+          Offset = BC.MIB->getAnnotationWithDefault<uint32_t>(II, "Offset",
+                                                              InvalidOffset);
+          assert(Offset != InvalidOffset && "Invalid offset");
+        }
+
+        if (Offset < InlinedCall.ParentPc) {
+          Offset += BC.computeInstructionSize(II, Emitter);
+          continue;
+        }
+
+        assert(Offset == InlinedCall.ParentPc && "Offset overflow");
+
+        // NOTE Annotatations must not be created with in concurent threads
+        static std::atomic_flag Lock = ATOMIC_FLAG_INIT;
+        while (Lock.test_and_set(std::memory_order_acquire))
+          ;
+        addFuncdataAnnotation(BC, II, Index, I, AllocId);
+        Lock.clear(std::memory_order_release);
+        // To be able to restore right inline unwinding we will lock the
+        // instruction
+        bool &Locked = BC.MIB->getOrCreateAnnotationAs<bool>(II, "Locked");
+        Locked = true;
+        break;
+      }
+
+      break;
+    }
+  }
+}
+
+static uint32_t readVarintPass(BinaryFunction *Function, GoFunc *GoFunc,
+                               DataExtractor &DE, uint64_t *MapOffset,
+                               const uint32_t Index, const uint8_t Quantum,
+                               const unsigned AllocId,
+                               const MCCodeEmitter *Emitter) {
+  BinaryContext &BC = Function->getBinaryContext();
+  int32_t ValSum = -1, MaxVal = -1;
+  uint64_t OldOffsetSum, OffsetSum = 0;
+  bool IsFirst = true;
+  MCInst *PrevII = nullptr;
+
+  do {
+    OldOffsetSum = OffsetSum;
+    int32_t Val = readVarintPair(DE, MapOffset, ValSum, OffsetSum, Quantum);
+    if (!Val && !IsFirst) {
+      if (Index == GoFunc->getPcdataStackMapIndex())
+        addVarintAnnotation(BC, *PrevII, Index, ValSum, /*IsNext*/ true,
+                            AllocId);
+      break;
+    }
+
+    if (ValSum > MaxVal)
+      MaxVal = ValSum;
+
+    for (BinaryBasicBlock *BB : Function->getLayout().blocks()) {
+      if (OldOffsetSum >= BB->getEndOffset())
+        continue;
+
+      uint32_t Offset = BB->getOffset();
+      if (Offset > OffsetSum)
+        break;
+
+      for (MCInst &II : *BB) {
+        if (BC.MIB->hasAnnotation(II, "Offset")) {
+          constexpr size_t InvalidOffset = std::numeric_limits<uint32_t>::max();
+          Offset = BC.MIB->getAnnotationWithDefault<uint32_t>(II, "Offset",
+                                                              InvalidOffset);
+          assert(Offset != InvalidOffset && "Invalid offset");
+        }
+
+        if (Offset < OldOffsetSum) {
+          Offset += BC.computeInstructionSize(II, Emitter);
+          continue;
+        } else if (Offset == OffsetSum) {
+          break;
+        }
+
+        addVarintAnnotation(BC, II, Index, ValSum, /*IsNext*/ false, AllocId);
+        if (Index == GoFunc->getPcdataStackMapIndex() && PrevII)
+          addVarintAnnotation(BC, *PrevII, Index, ValSum, /*IsNext*/ true,
+                              AllocId);
+
+        PrevII = &II;
+        assert(Offset < OffsetSum && "Offset overflow");
+        Offset += BC.computeInstructionSize(II, Emitter);
+      }
+    }
+
+    IsFirst = false;
+  } while (1);
+
+  return MaxVal;
+}
+
+void GolangPrePass::deferreturnPass(BinaryFunction &BF,
+                                    const uint64_t DeferOffset,
+                                    const unsigned AllocId,
+                                    const MCCodeEmitter *Emitter) {
+  BinaryContext &BC = BF.getBinaryContext();
+  uint64_t Offset = 0;
+  for (BinaryBasicBlock *BB : BF.getLayout().blocks()) {
+    for (MCInst &II : *BB) {
+      if (BC.MIB->hasAnnotation(II, "Offset")) {
+        constexpr auto InvalidOffset = std::numeric_limits<uint32_t>::max();
+        Offset = BC.MIB->getAnnotationWithDefault<uint32_t>(II, "Offset",
+                                                            InvalidOffset);
+        assert(Offset != InvalidOffset);
+      }
+
+      if (Offset < DeferOffset) {
+        Offset += BC.computeInstructionSize(II, Emitter);
+        continue;
+      }
+
+      if (Offset != DeferOffset)
+        break;
+
+      assert(BC.MIB->isCall(II));
+      BC.MIB->addAnnotation(II, "IsDefer", true, AllocId);
+      return;
+    }
+  }
+
+  outs() << "Deferreturn call was not found for " << BF << "\n";
+  exit(1);
+}
+
+int GolangPrePass::pclntabPass(BinaryContext &BC) {
+  const uint64_t PclntabAddr = getPcHeaderAddr();
+  if (!PclntabAddr) {
+    errs() << "BOLT-ERROR: Pclntab address is zero!\n";
+    return -1;
+  }
+
+  BinaryData *PclntabSym = BC.getBinaryDataAtAddress(PclntabAddr);
+  if (!PclntabSym) {
+    errs() << "BOLT-ERROR: Failed to get pclntab symbol!\n";
+    return -1;
+  }
+
+  BinarySection *Section = &PclntabSym->getSection();
+  const class Pclntab *Pclntab = getPclntab();
+  uint64_t Offset = Pclntab->getPclntabOffset();
+  DataExtractor DE = DataExtractor(Section->getContents(),
+                                   BC.AsmInfo->isLittleEndian(), getPsize());
+  for (uint64_t F = 0; F < Pclntab->getFunctionsCount(); ++F) {
+    assert(DE.isValidOffset(Offset) && "Invalid offset");
+    struct Functab Functab;
+    RemoveRelaReloc(BC, Section, Offset);
+    Functab.Address = DE.getAddress(&Offset);
+    Functab.Offset = DE.getAddress(&Offset);
+
+    BinaryFunction *Function = BC.getBinaryFunctionAtAddress(Functab.Address);
+    if (!Function) {
+      outs() << "Failed to find function by address "
+             << Twine::utohexstr(Functab.Address) << "\n";
+      return -1;
+    }
+
+    Function->setGolangFunctabOffset(Pclntab->getFunctabOffset() +
+                                     Functab.Offset);
+  }
+
+  // Remove maxpc relocation (last pclntab entry)
+  RemoveRelaReloc(BC, Section, Offset);
+
+  ParallelUtilities::WorkFuncWithAllocTy WorkFun =
+      [&](BinaryFunction &Function, MCPlusBuilder::AllocatorIdTy AllocId) {
+        if (Function.getLayout().block_begin() ==
+            Function.getLayout().block_end())
+          return;
+
+        BinaryContext::IndependentCodeEmitter Emitter;
+        if (!opts::NoThreads) {
+          Emitter =
+              Function.getBinaryContext().createIndependentMCCodeEmitter();
+        }
+
+        uint64_t FuncOffset = Function.getGolangFunctabOffset();
+        std::unique_ptr<struct GoFunc> GoFunc = createGoFunc();
+        GoFunc->read(BC, DE, Section, &FuncOffset);
+        if (GoFunc->hasReservedID(Function.getDemangledName())) {
+          // The functions with reserved ID are special functions
+          // mostly written on asm that are dangerous to change
+          Function.setSimple(false);
+        }
+
+        // Special functions that we must not change
+        const std::unordered_set<std::string> SimpleFuncs = {
+            "runtime.gcWriteBarrier", "runtime.duffzero", "runtime.duffcopy"};
+
+        if (SimpleFuncs.find(Function.getDemangledName()) != SimpleFuncs.end())
+          Function.setSimple(false);
+
+        auto GetPcdata = [&](const uint32_t Index) -> bool {
+          int32_t Max = -1;
+          uint32_t MapOffsetVal = GoFunc->getPcdata(Index);
+          if (MapOffsetVal) {
+            uint64_t MapOffset = Pclntab->getPctabOffset() + MapOffsetVal;
+            Max = readVarintPass(&Function, GoFunc.get(), DE, &MapOffset, Index,
+                                 Pclntab->getQuantum(), AllocId,
+                                 Emitter.MCE.get());
+          }
+
+          GoFunc->setPcdataMaxVal(Index, Max);
+          return !!MapOffsetVal;
+        };
+
+        if (!GetPcdata(GoFunc->getPcdataUnsafePointIndex())) {
+          // The function has no PCDATA_UnsafePoint info, so we will mark every
+          // instruction as a safe one.
+          const int SafePoint = GoFunc->getPcdataSafePointVal();
+          const uint32_t Index = GoFunc->getPcdataUnsafePointIndex();
+          for (BinaryBasicBlock *BB : Function.getLayout().blocks())
+            for (auto II = BB->begin(); II != BB->end(); ++II)
+              addVarintAnnotation(BC, *II, Index, SafePoint, /*IsNext*/ false,
+                                  AllocId);
+        }
+
+        GetPcdata(GoFunc->getPcdataStackMapIndex());
+        GetPcdata(GoFunc->getPcdataInlTreeIndex());
+
+        uint64_t DeferOffset = GoFunc->getDeferreturnOffset();
+        if (DeferOffset)
+          deferreturnPass(Function, DeferOffset, AllocId, Emitter.MCE.get());
+
+        // If the function does not have stack map index varint
+        // it was probably written in asm
+        if (GoFunc->getPcdataMax(GoFunc->getPcdataStackMapIndex()) == -1)
+          Function.setIsAsm(true);
+
+        // ASM Functions might use the system stack and we won't be able to
+        // locate that the stack was switched.
+        // TODO For functions with deferreturn calls we preserve the table since
+        // the BB is unreachable we are unable calculate stack offset currently.
+        if (GoFunc->getPcspOffset() &&
+            (Function.isAsm() || DeferOffset || opts::GolangPcspPreserve)) {
+          uint64_t Offset = Pclntab->getPctabOffset() + GoFunc->getPcspOffset();
+          readVarintPass(&Function, GoFunc.get(), DE, &Offset,
+                         GoFunc->getPcspIndex(), Pclntab->getQuantum(), AllocId,
+                         Emitter.MCE.get());
+        }
+
+        {
+          // Remove funcdata relocations
+          uint32_t Foffset = GoFunc->getFuncdataOffset();
+          for (int I = 0; I < GoFunc->getNfuncdata(); ++I) {
+            RemoveRelaReloc(BC, Section, Foffset);
+            Foffset += getPsize();
+          }
+        }
+
+        inlTreePass(&Function, GoFunc.get(), AllocId, Emitter.MCE.get());
+      };
+
+  ParallelUtilities::PredicateTy skipFunc =
+      [&](const BinaryFunction &Function) { return !Function.isGolang(); };
+
+  ParallelUtilities::runOnEachFunctionWithUniqueAllocId(
+      BC, ParallelUtilities::SchedulingPolicy::SP_INST_QUADRATIC, WorkFun,
+      skipFunc, "pcdataGoPreProcess", /*ForceSequential*/ true);
+
+  return 0;
+}
+
+void GolangPrePass::goPassInit(BinaryContext &BC) {
+  // NOTE Currently we don't support PCSP table restoration for
+  // AARCH64 since we have many ways to increment/decrement stack
+  // values and often stack value is changed through other
+  // registers so we will need to track all registers in order
+  // to properly find stack movement values
+  if (BC.isAArch64() && !opts::GolangPcspPreserve) {
+    LLVM_DEBUG(
+        dbgs() << "BOLT-INFO: Enabling GolangPcspPreserve for AARCH64!\n");
+    opts::GolangPcspPreserve = true;
+  }
+
+  BC.MIB->getOrCreateAnnotationIndex("IsDefer");
+
+  // Initialize annotation index for multi-thread access
+  std::unique_ptr<struct GoFunc> GoFunc = createGoFunc();
+  auto initAnnotation = [&](const unsigned Index) {
+    BC.MIB->getOrCreateAnnotationIndex(getVarintName(Index));
+    BC.MIB->getOrCreateAnnotationIndex(getVarintName(Index, /*IsNext*/ true));
+  };
+
+  initAnnotation(GoFunc->getPcdataUnsafePointIndex());
+  initAnnotation(GoFunc->getPcdataStackMapIndex());
+  initAnnotation(GoFunc->getPcdataInlTreeIndex());
+  initAnnotation(GoFunc->getPcspIndex());
+}
+
+void GolangPrePass::nopPass(BinaryContext &BC) {
+  // The golang might gemerate unreachable jumps e.g.
+  // https://go-review.googlesource.com/c/go/+/380894/
+  // Removing the nops at branch destination might affect PCSP table generation
+  // for the code below the nop. Remove NOP instruction annotation at the
+  // beginning of the basic block in order to preserve BB layout for such cases.
+  // Shorten multi-byte NOP before annotation remove.
+
+  ParallelUtilities::WorkFuncWithAllocTy WorkFun =
+      [&](BinaryFunction &Function, MCPlusBuilder::AllocatorIdTy AllocId) {
+        for (BinaryBasicBlock *BB : Function.getLayout().blocks()) {
+          MCInst &Inst = BB->front();
+          if (!BC.MIB->isNoop(Inst))
+            continue;
+
+          BC.MIB->shortenInstruction(Inst, *BC.STI);
+          BC.MIB->removeAnnotation(Inst, "NOP");
+          BC.MIB->removeAnnotation(Inst, "Size");
+        }
+      };
+
+  ParallelUtilities::PredicateTy skipFunc =
+      [&](const BinaryFunction &Function) { return !Function.isGolang(); };
+
+  ParallelUtilities::runOnEachFunctionWithUniqueAllocId(
+      BC, ParallelUtilities::SchedulingPolicy::SP_INST_QUADRATIC, WorkFun,
+      skipFunc, "nopGoPreProcess", /*ForceSequential*/ true);
+}
+
+void GolangPrePass::runOnFunctions(BinaryContext &BC) {
+  int Ret;
+
+  goPassInit(BC);
+
+  Ret = pclntabPass(BC);
+  if (Ret < 0) {
+    errs() << "BOLT-ERROR: Golang preprocess pclntab pass failed!\n";
+    exit(1);
+  }
+
+  nopPass(BC);
+}
+
+} // end namespace bolt
+} // end namespace llvm

bolt/lib/Passes/Golang.cpp

+//===--- Golang.cpp -------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+//===----------------------------------------------------------------------===//
+
+#include "bolt/Passes/Golang.h"
+#include "bolt/Core/ParallelUtilities.h"
+#include "llvm/Support/EndianStream.h"
+
+#define DEBUG_TYPE "bolt-golang"
+
+using namespace llvm;
+using namespace bolt;
+
+namespace opts {
+extern cl::OptionCategory BoltOptCategory;
+
+extern cl::opt<bool> Instrument;
+extern cl::opt<bool> NoHugePages;
+extern cl::opt<unsigned> AlignFunctions;
+
+cl::opt<bool>
+    GolangPcspPreserve("golang-preserve-pcsp",
+                       cl::desc("Save pcsp table instead of reconstructing it"),
+                       cl::init(false), cl::ZeroOrMore, cl::Hidden,
+                       cl::cat(BoltOptCategory));
+
+} // end namespace opts
+
+namespace llvm {
+namespace bolt {
+
+#define KINDMASK ((1 << 5) - 1) // runtime/typekind.go
+#define UNCOMMON_FLAG (1 << 0)  // runtime/type.go
+
+// reflect/type.go ; runtime/typekind.go
+enum Kind {
+  Invalid = 0,
+  Bool,
+  Int,
+  Int8,
+  Int16,
+  Int32,
+  Int64,
+  Uint,
+  Uint8,
+  Uint16,
+  Uint32,
+  Uint64,
+  Uintptr,
+  Float32,
+  Float64,
+  Complex64,
+  Complex128,
+  Array,
+  Chan,
+  Func,
+  Interface,
+  Map,
+  Ptr,
+  Slice,
+  String,
+  Struct,
+  UnsafePointer,
+  LastKind
+};
+
+static std::map<const BinaryBasicBlock *, uint64_t> BBSizes;
+
+static int updateBBSizes(BinaryContext &BC) {
+  ParallelUtilities::WorkFuncTy WorkFun = [&](BinaryFunction &BF) {
+    for (BinaryBasicBlock *BB : BF.getLayout().blocks())
+      BBSizes[BB] = BB->estimateSize();
+  };
+
+  ParallelUtilities::runOnEachFunction(
+      BC, ParallelUtilities::SchedulingPolicy::SP_TRIVIAL, WorkFun, nullptr,
+      "UpdateEstimatedSizes",
+      /*ForceSequential*/ true);
+  return 0;
+}
+
+static uint64_t getBFCodeSize(BinaryFunction *BF) {
+  uint64_t Size = 0;
+  for (BinaryBasicBlock *BB : BF->getLayout().blocks())
+    Size += BBSizes[BB];
+
+  return Size;
+}
+
+static uint64_t getBFSize(BinaryFunction *BF) {
+  uint64_t Size = getBFCodeSize(BF);
+  Size += BF->estimateConstantIslandSize();
+  return Size;
+}
+
+static uint64_t getBFInstrOffset(const BinaryBasicBlock *BB,
+                                 const MCInst *Instr) {
+  uint64_t Offset = 0;
+  BinaryFunction *BF = BB->getFunction();
+  for (BinaryBasicBlock *BasicBlock : BF->getLayout().blocks()) {
+    if (BB == BasicBlock)
+      break;
+
+    Offset += BBSizes[BasicBlock];
+  }
+
+  BinaryContext &BC = BF->getBinaryContext();
+  for (const MCInst &II : *BB) {
+    if (Instr == &II)
+      return Offset;
+
+    Offset += BC.computeInstructionSize(II);
+  }
+
+  llvm_unreachable("Wrong BB or Instr");
+  exit(1);
+}
+
+inline static uint64_t getNewTextStart(BinaryContext &BC) {
+  // NOTE The new text address is allocated after all passes were finished,
+  // for now use the first free address stored in BC.LayoutStartAddress
+  return alignTo(BC.LayoutStartAddress, BC.PageAlign);
+}
+
+BinaryFunction *getBF(BinaryContext &BC, std::vector<BinaryFunction *> &BFs,
+                      const char *Name) {
+  for (auto BFit = BFs.rbegin(); BFit != BFs.rend(); ++BFit) {
+    BinaryFunction *BF = *BFit;
+    if (BF->hasRestoredNameRegex(Name))
+      return BF;
+  }
+
+  return nullptr;
+}
+
+BinaryFunction *getFirstBF(BinaryContext &BC,
+                           std::vector<BinaryFunction *> &BFs) {
+  return getBF(BC, BFs, GolangPass::getFirstBFName());
+}
+
+BinaryFunction *getLastBF(BinaryContext &BC,
+                          std::vector<BinaryFunction *> &BFs) {
+  return getBF(BC, BFs, GolangPass::getLastBFName());
+}
+
+uint64_t readEndianVal(DataExtractor &DE, uint64_t *Offset, uint16_t Size);
+
+uint32_t readVarint(DataExtractor &DE, uint64_t *Offset);
+
+int32_t readVarintPair(DataExtractor &DE, uint64_t *MapOffset, int32_t &ValSum,
+                       uint32_t &OffsetSum);
+
+static void writeVarint(uint8_t *Data, uint64_t *Offset, uint32_t Val) {
+  while (Val >= 0x80) {
+    Data[(*Offset)++] = (uint8_t)(Val | 0x80);
+    Val >>= 7;
+  }
+
+  Data[(*Offset)++] = (uint8_t)Val;
+}
+
+static void writeVarint(uint8_t **Data, uint32_t Val) {
+  uint64_t Offset = 0;
+  writeVarint(*Data, &Offset, Val);
+  *Data += Offset;
+}
+
+static void writeVarintPair(int32_t Val, int32_t &PrevVal, uint64_t Offset,
+                            uint64_t &CurrentOffset, bool &IsFirst,
+                            uint8_t **DataFuncOffset, const uint8_t Quantum) {
+  int32_t V = Val - PrevVal;
+  V = (V < 0) ? (((-V - 1) << 1) | 1) : V << 1;
+  assert((V != 0 || IsFirst) && "The value detla could not be zero");
+  PrevVal = Val;
+  writeVarint(DataFuncOffset, (uint32_t)V);
+  assert((Offset - CurrentOffset) % Quantum == 0 &&
+         "Offset it not multiple of quantum");
+  uint32_t CurrentDelta = (Offset - CurrentOffset) / Quantum;
+  assert((CurrentDelta || IsFirst) && "The offset delta could not be zero");
+  writeVarint(DataFuncOffset, CurrentDelta);
+  CurrentOffset = Offset;
+  IsFirst = false;
+}
+
+template <typename T> static T writeEndian(BinaryContext &BC, T Val) {
+  T Ret;
+  SmallVector<char, sizeof(T)> Tmp;
+  raw_svector_ostream OS(Tmp);
+  enum support::endianness Endian = support::big;
+  if (BC.AsmInfo->isLittleEndian())
+    Endian = support::little;
+
+  struct support::endian::Writer Writer(OS, Endian);
+  Writer.write<T>(Val);
+  memcpy(&Ret, OS.str().data(), sizeof(T));
+  return Ret;
+}
+
+void writeEndianVal(BinaryContext &BC, uint64_t Val, uint16_t Size,
+                    uint8_t **Res) {
+  switch (Size) {
+  case 8: {
+    uint64_t Endian = writeEndian<uint64_t>(BC, Val);
+    **(uint64_t **)Res = Endian;
+    break;
+  }
+
+  case 4: {
+    uint32_t Endian = writeEndian<uint32_t>(BC, (uint32_t)Val);
+    **(uint32_t **)Res = Endian;
+    break;
+  }
+
+  case 2: {
+    uint16_t Endian = writeEndian<uint16_t>(BC, (uint16_t)Val);
+    **(uint16_t **)Res = Endian;
+    break;
+  }
+
+  case 1: {
+    **Res = (uint8_t)Val;
+    break;
+  }
+
+  default:
+    llvm_unreachable("Wrong type size");
+    exit(1);
+  }
+
+  *Res += Size;
+}
+
+inline void writeEndianPointer(BinaryContext &BC, uint64_t Val, uint8_t **Res) {
+  return writeEndianVal(BC, Val, BC.AsmInfo->getCodePointerSize(), Res);
+}
+
+std::string getVarintName(uint32_t Index, bool IsNext);
+
+void addVarintAnnotation(BinaryContext &BC, MCInst &II, uint32_t Index,
+                         int32_t Value, bool IsNext, unsigned AllocId);
+
+bool hasVarintAnnotation(BinaryContext &BC, MCInst &II, uint32_t Index,
+                         bool IsNext);
+
+int32_t getVarintAnnotation(BinaryContext &BC, MCInst &II, uint32_t Index,
+                            bool IsNext);
+
+std::string getFuncdataName(uint32_t Findex, uint32_t Size);
+
+std::string getFuncdataSizeName(uint32_t Findex);
+
+void addFuncdataAnnotation(BinaryContext &BC, MCInst &II, uint32_t Findex,
+                           int32_t Value, unsigned AllocId);
+
+bool hasFuncdataAnnotation(BinaryContext &BC, MCInst &II, uint32_t Findex);
+
+uint32_t getFuncdataSizeAnnotation(BinaryContext &BC, MCInst &II,
+                                   uint32_t Findex);
+
+int32_t getFuncdataAnnotation(BinaryContext &BC, MCInst &II, uint32_t Findex,
+                              uint32_t Index);
+
+void RemoveRelaReloc(BinaryContext &BC, BinarySection *Section,
+                     uint64_t Offset);
+
+void AddRelaReloc(BinaryContext &BC, MCSymbol *Symbol, BinarySection *Section,
+                  uint64_t Offset, uint64_t Addend = 0);
+
+static std::vector<BinaryFunction *>
+getSortedGolangFunctions(BinaryContext &BC) {
+  std::vector<BinaryFunction *> BFs = BC.getSortedFunctions();
+  BFs.erase(std::remove_if(BFs.begin(), BFs.end(),
+                           [](BinaryFunction *BF) {
+                             return !BF->isGolang() || BF->isFolded();
+                           }),
+            BFs.end());
+  return BFs;
+}
+
+Pclntab::~Pclntab() {}
+
+int Pclntab::readHeader(BinaryContext &BC, const uint64_t PclntabHeaderAddr) {
+  BinaryData *PclntabSym = BC.getBinaryDataAtAddress(PclntabHeaderAddr);
+  if (!PclntabSym) {
+    errs() << "BOLT-ERROR: failed to get pclntab symbol!\n";
+    return -1;
+  }
+
+  BinarySection *Section = &PclntabSym->getSection();
+  uint64_t Offset = PclntabHeaderAddr - Section->getAddress();
+  setPclntabHeaderOffset(Offset);
+  DataExtractor DE =
+      DataExtractor(Section->getContents(), BC.AsmInfo->isLittleEndian(),
+                    BC.AsmInfo->getCodePointerSize());
+
+  __readHeader(BC, DE);
+
+  if (!checkMagic()) {
+    errs() << "BOLT-ERROR: pclntab bad magic!\n";
+    return -1;
+  }
+
+  if (getPsize() != BC.AsmInfo->getCodePointerSize()) {
+    outs() << "BOLT-ERROR: pclntab bad pointer size!\n";
+    return -1;
+  }
+
+  return 0;
+}
+
+int Pclntab::writeHeader(BinaryContext &BC, uint8_t *Pclntab) {
+  setNewHeaderOffsets();
+  __writeHeader(BC, Pclntab);
+  return 0;
+}
+
+Module::~Module() {}
+
+int Module::read(const BinaryContext &BC) {
+  // NOTE The local.moduledata are used in plugins.
+  // The firstmoduledata symbol still could be found there
+  // but it will point in BSS section */
+  const BinaryData *Module = BC.getFirstBinaryDataByName("local.moduledata");
+  if (!Module)
+    Module = BC.getFirstBinaryDataByName("runtime.firstmoduledata");
+
+  if (!Module) {
+    errs() << "BOLT-ERROR: failed to get firstmoduledata symbol!\n";
+    return -1;
+  }
+
+  const BinarySection *Section = &Module->getSection();
+  DataExtractor DE =
+      DataExtractor(Section->getContents(), BC.AsmInfo->isLittleEndian(),
+                    BC.AsmInfo->getCodePointerSize());
+
+  uint64_t Offset = Module->getAddress() - Section->getAddress();
+  uint64_t *ModuleArr = getModule();
+  for (size_t I = 0; I < getModuleSize() / sizeof(uint64_t); ++I) {
+    assert(DE.isValidOffset(Offset) && "Invalid offset");
+    ModuleArr[I] = DE.getAddress(&Offset);
+  }
+
+  return 0;
+}
+
+GoFunc::~GoFunc() {}
+
+std::unique_ptr<struct Module> createGoModule() {
+  if (opts::GolangPass == opts::GV_1_17_8)
+    return std::make_unique<Module_v1_17_8>();
+  else if (opts::GolangPass == opts::GV_1_17_5)
+    return std::make_unique<Module_v1_17_5>();
+  else if (opts::GolangPass == opts::GV_1_17_2)
+    return std::make_unique<Module_v1_17_2>();
+  else if (opts::GolangPass == opts::GV_1_16_5)
+    return std::make_unique<Module_v1_16_5>();
+  else if (opts::GolangPass == opts::GV_1_14_12)
+    return std::make_unique<Module_v1_14_12>();
+  else if (opts::GolangPass == opts::GV_1_14_9)
+    return std::make_unique<Module_v1_14_9>();
+
+  llvm_unreachable("Wrong golang version");
+  exit(1);
+}
+
+std::unique_ptr<class Pclntab> createGoPclntab() {
+  if (opts::GolangPass == opts::GV_1_17_8)
+    return std::make_unique<Pclntab_v1_17_8>();
+  else if (opts::GolangPass == opts::GV_1_17_5)
+    return std::make_unique<Pclntab_v1_17_5>();
+  else if (opts::GolangPass == opts::GV_1_17_2)
+    return std::make_unique<Pclntab_v1_17_2>();
+  else if (opts::GolangPass == opts::GV_1_16_5)
+    return std::make_unique<Pclntab_v1_16_5>();
+  else if (opts::GolangPass == opts::GV_1_14_12)
+    return std::make_unique<Pclntab_v1_14_12>();
+  else if (opts::GolangPass == opts::GV_1_14_9)
+    return std::make_unique<Pclntab_v1_14_9>();
+
+  llvm_unreachable("Wrong golang version");
+  exit(1);
+}
+
+std::unique_ptr<struct GoFunc> createGoFunc() {
+  if (opts::GolangPass == opts::GV_1_17_8)
+    return std::make_unique<GoFunc_v1_17_8>();
+  else if (opts::GolangPass == opts::GV_1_17_5)
+    return std::make_unique<GoFunc_v1_17_5>();
+  else if (opts::GolangPass == opts::GV_1_17_2)
+    return std::make_unique<GoFunc_v1_17_2>();
+  else if (opts::GolangPass == opts::GV_1_16_5)
+    return std::make_unique<GoFunc_v1_16_5>();
+  else if (opts::GolangPass == opts::GV_1_14_12)
+    return std::make_unique<GoFunc_v1_14_12>();
+  else if (opts::GolangPass == opts::GV_1_14_9)
+    return std::make_unique<GoFunc_v1_14_9>();
+
+  llvm_unreachable("Wrong golang version");
+  exit(1);
+}
+
+struct StackVal {
+  uint32_t Size;
+  int32_t OldVal;
+  int32_t Val;
+};
+
+using InstBias = std::map<uint32_t, struct StackVal>;
+
+static uint32_t stackCounter(BinaryFunction *BF, BinaryBasicBlock *BB,
+                             InstBias &Map, uint32_t SpVal) {
+  BinaryContext &BC = BF->getBinaryContext();
+  unsigned pSize = BC.AsmInfo->getCodePointerSize();
+  auto addVal = [&](InstBias &Map, MCInst *II, int32_t OldVal, int32_t NewVal) {
+    struct StackVal StackVal;
+    uint32_t Offset = getBFInstrOffset(BB, II);
+    StackVal.Size = BC.computeInstructionSize(*II);
+    StackVal.OldVal = OldVal;
+    StackVal.Val = NewVal;
+    Map[Offset] = StackVal;
+  };
+
+  for (MCInst &II : *BB) {
+    int Ret = 0;
+    if (&II == &(*BB->begin()) || &II == &(*BB->rbegin()))
+      addVal(Map, &II, SpVal, SpVal);
+
+    // Ignore instrumentation stack usage
+    if (opts::Instrument && BC.MIB->hasAnnotation(II, "IsInstrumentation"))
+      continue;
+
+    if ((Ret = BC.MIB->getStackAdjustment(II))) {
+      // NOTE Seems to be the only exception is runtime.rt0_go function
+      if (std::abs(Ret) % pSize == 0) {
+        addVal(Map, &II, SpVal, SpVal + Ret);
+        SpVal += Ret;
+      }
+    }
+  }
+
+  return SpVal;
+}
+
+int GolangPass::typePass(BinaryContext &BC, uint64_t TypeAddr) {
+  static std::unordered_set<uint64_t> VisitedTypes;
+  uint64_t Offset;
+  uint64_t SectionAddr;
+
+  if (VisitedTypes.find(TypeAddr) != VisitedTypes.end())
+    return 0;
+
+  VisitedTypes.insert(TypeAddr);
+  ErrorOr<BinarySection &> Section = BC.getSectionForAddress(TypeAddr);
+  if (!Section) {
+    errs() << "BOLT-ERROR: failed to get section for type 0x"
+           << Twine::utohexstr(TypeAddr) << "\n";
+    return -1;
+  }
+
+  SectionAddr = Section->getAddress();
+  DataExtractor DE = DataExtractor(Section->getContents(),
+                                   BC.AsmInfo->isLittleEndian(), getPsize());
+  Offset = TypeAddr - SectionAddr;
+  assert(DE.isValidOffset(Offset) && "Invalid offset");
+
+  // runtime/type.go
+  struct Type {
+    uint64_t Size;    // Pointer size
+    uint64_t Ptrdata; // Pointer size
+    uint32_t Hash;
+    uint8_t Tflag;
+    uint8_t Align;
+    uint8_t Fieldalign;
+    uint8_t Kind;
+    uint64_t CompareFunc; // Pointer size
+    uint64_t Gcdata;      // Pointer size
+    int32_t NameOff;
+    int32_t PtrToThis;
+  } Type;
+
+  Type.Size = DE.getAddress(&Offset);
+  Type.Ptrdata = DE.getAddress(&Offset);
+  Type.Hash = DE.getU32(&Offset);
+  Type.Tflag = DE.getU8(&Offset);
+  Type.Align = DE.getU8(&Offset);
+  Type.Fieldalign = DE.getU8(&Offset);
+  Type.Kind = DE.getU8(&Offset);
+  Type.CompareFunc = DE.getAddress(&Offset);
+  Type.Gcdata = DE.getAddress(&Offset);
+  Type.NameOff = (int32_t)DE.getU32(&Offset);
+  Type.PtrToThis = (int32_t)DE.getU32(&Offset);
+
+  if (!(Type.Tflag & UNCOMMON_FLAG))
+    return 0;
+
+  uint8_t Kind = Type.Kind & KINDMASK;
+  assert(Kind < LastKind && "Wrong kind type");
+  assert(DE.isValidOffset(Offset) && "Wrong offset");
+
+  // The furter structures are in runtime/type.go file
+  if ((Kind == Ptr) || (Kind == Slice)) {
+    // struct ptrtype {
+    //   //typ  _type;
+    //   struct Type *elem;
+    // };
+    //
+    // struct slicetype {
+    //   //typ  _type;
+    //   elem *_type;
+    // };
+
+    uint64_t Address = DE.getAddress(&Offset);
+    typePass(BC, Address);
+  } else if (Kind == Struct) {
+    // struct structtype {
+    //   //struct Type typ;
+    //   pkgPath name // bytes *byte;
+    //   fields  []structfield;
+    // };
+
+    struct {
+      uint64_t Bytes;      // Pointer size
+      uint64_t Type;       // Pointer size
+      uint64_t OffsetAnon; // Pointer size
+    } Structfield;
+
+    Offset += getPsize(); // Skip Name
+    uint64_t StructfieldAddress = DE.getAddress(&Offset);
+    uint64_t Size = DE.getAddress(&Offset);
+    Offset += getPsize(); // Skip second size
+
+    assert(Section->containsAddress(StructfieldAddress) &&
+           "Wrong StructfieldAddress");
+    uint64_t StructfieldOffset = StructfieldAddress - Section->getAddress();
+    while (Size--) {
+      Structfield.Bytes = DE.getAddress(&StructfieldOffset);
+      Structfield.Type = DE.getAddress(&StructfieldOffset);
+      Structfield.OffsetAnon = DE.getAddress(&StructfieldOffset);
+      typePass(BC, Structfield.Type);
+    }
+  } else if (Kind == Interface) {
+    // struct interfacetype {
+    //  //struct Type typ;
+    //  pkgPath name // bytes *byte;
+    //  mhdr    []Imethod;
+    // };
+
+    struct {
+      int32_t Name;
+      int32_t Ityp;
+    } Imethod;
+
+    Offset += getPsize(); // Skip Name
+    uint64_t MhdrAddress = DE.getAddress(&Offset);
+    uint64_t Size = DE.getAddress(&Offset);
+    Offset += getPsize(); // Skip second size
+
+    assert(Section->containsAddress(MhdrAddress) && "Wrong MhdrAddress");
+    uint64_t MhdrOffset = MhdrAddress - Section->getAddress();
+    while (Size--) {
+      Imethod.Name = (int32_t)DE.getU32(&MhdrOffset);
+      Imethod.Ityp = (int32_t)DE.getU32(&MhdrOffset);
+      typePass(BC, FirstModule->getTypes() + Imethod.Ityp);
+    }
+  } else if (Kind == Func) {
+    // struct functype {
+    //  //typ      _type;
+    //  inCount  uint16;
+    //  outCount uint16;
+    // };
+
+    Offset += 2 * sizeof(uint16_t);
+    // NOTE in this case we must align offset
+    Offset = alignTo(Offset, getPsize());
+  } else if (Kind == Array) {
+    // struct arraytype {
+    //   //typ   _type;
+    //   elem  *_type;
+    //   slice *_type;
+    //   len   uintptr;
+    // };
+
+    uint64_t Addr;
+    Addr = DE.getAddress(&Offset);
+    typePass(BC, Addr);
+    Addr = DE.getAddress(&Offset);
+    typePass(BC, Addr);
+    Offset += getPsize();
+  } else if (Kind == Chan) {
+    // struct chantype {
+    //  //typ  _type;
+    //  elem *_type;
+    //  dir  uintptr;
+    // };
+
+    uint64_t Addr = DE.getAddress(&Offset);
+    typePass(BC, Addr);
+    Offset += getPsize();
+  } else if (Kind == Map) {
+    // Large structure, seems to be no align needed though
+    // The first 3 fields are type struct
+    uint64_t Addr;
+    Addr = DE.getAddress(&Offset);
+    typePass(BC, Addr);
+    Addr = DE.getAddress(&Offset);
+    typePass(BC, Addr);
+    Addr = DE.getAddress(&Offset);
+    typePass(BC, Addr);
+    Offset += 2 * getPsize();
+  }
+
+  assert(Offset == alignTo(Offset, getPsize()) && "Wrong alignment");
+  assert(DE.isValidOffset(Offset) && "Invalid Offset");
+
+  uint64_t UncommonOffset = Offset;
+
+  // runtime/type.go
+  struct {
+    int32_t Pkgpath;
+    uint16_t Mcount;
+    uint16_t Xcount;
+    uint32_t Moff;
+    uint32_t Unused2;
+  } Uncommontype;
+
+  Uncommontype.Pkgpath = (int32_t)DE.getU32(&Offset);
+  Uncommontype.Mcount = DE.getU16(&Offset);
+  Uncommontype.Xcount = DE.getU16(&Offset);
+  Uncommontype.Moff = DE.getU32(&Offset);
+  Uncommontype.Unused2 = DE.getU32(&Offset);
+
+  assert(UncommonOffset + Uncommontype.Moff >= Offset && "Wrong Moff");
+  Offset = UncommonOffset + Uncommontype.Moff;
+  while (Uncommontype.Mcount--) {
+    assert(DE.isValidOffset(Offset) && "Invalid offset");
+    // runtime/type.go
+    struct {
+      int32_t NameOff;
+      int32_t TypeOff;
+      int32_t Ifn;
+      int32_t Tfn;
+    } Method;
+
+    Method.NameOff = (int32_t)DE.getU32(&Offset);
+    Method.TypeOff = (int32_t)DE.getU32(&Offset);
+    uint32_t IfnOffset = Offset;
+    Method.Ifn = (int32_t)DE.getU32(&Offset);
+    uint32_t TfnOffset = Offset;
+    Method.Tfn = (int32_t)DE.getU32(&Offset);
+
+    uint64_t StartAddr = getNewTextStart(BC);
+    uint64_t RType = Relocation::getAbs(sizeof(uint32_t));
+    auto setFn = [&](int32_t Value, uint32_t Offset) {
+      if (Value == -1)
+        return;
+
+      BinaryFunction *Fn = BC.getBinaryFunctionAtAddress(RuntimeText + Value);
+      if (!Fn) {
+        errs() << "BOLT-ERROR: failed to get Ifn or Tfn!\n";
+        exit(1);
+      }
+
+      BC.addRelocation(SectionAddr + Offset, Fn->getSymbol(), RType,
+                       -StartAddr);
+    };
+
+    setFn(Method.Ifn, IfnOffset);
+    setFn(Method.Tfn, TfnOffset);
+  }
+
+  return 0;
+}
+
+int GolangPass::typelinksPass(BinaryContext &BC) {
+  int Ret;
+  uint64_t Types = FirstModule->getTypes();
+  if (!Types) {
+    errs() << "BOLT-ERROR: types address is zero!\n";
+    return -1;
+  }
+
+  uint64_t Etypes = FirstModule->getEtypes();
+  assert(Types < Etypes && "Wrong Etypes");
+  const GoArray &TypeLinks = FirstModule->getTypelinks();
+  uint64_t TypelinkAddr = TypeLinks.getAddress();
+  uint64_t TypelinkCount = TypeLinks.getCount();
+  if (!TypelinkAddr) {
+    errs() << "BOLT-ERROR: typelink address is zero!\n";
+    return -1;
+  }
+
+  ErrorOr<BinarySection &> Section = BC.getSectionForAddress(TypelinkAddr);
+  if (!Section) {
+    errs() << "BOLT-ERROR: failed to get typelink section!\n";
+    return -1;
+  }
+
+  DataExtractor DE = DataExtractor(Section->getContents(),
+                                   BC.AsmInfo->isLittleEndian(), getPsize());
+
+  uint64_t Offset = TypelinkAddr - Section->getAddress();
+  while (TypelinkCount--) {
+    assert(DE.isValidOffset(Offset) && "Invalid offset");
+    uint64_t Type = Types + DE.getU32(&Offset);
+    assert(Type < Etypes && "Wrong type offset");
+    Ret = typePass(BC, Type);
+    if (Ret < 0)
+      return Ret;
+  }
+
+  return 0;
+}
+
+int GolangPass::textsectmapPass(BinaryContext &BC) {
+  uint64_t TextSectAddr = FirstModule->getTextsectmap().getAddress();
+  if (!TextSectAddr) {
+    // Plugins does't have this structure
+    return 0;
+  }
+
+  ErrorOr<BinarySection &> Section = BC.getSectionForAddress(TextSectAddr);
+  if (!Section) {
+    errs() << "BOLT-ERROR: failed to get textsectmaps section!\n";
+    return -1;
+  }
+
+  BinaryData *EtextSymbol = BC.getFirstBinaryDataByName(getLastBFName());
+  if (!EtextSymbol) {
+    errs() << "BOLT-ERROR: failed to get etext symbol!\n";
+    return -1;
+  }
+
+  // We will need to fix length field (text size) of textsect structure
+  //
+  // //runtime/symtab.go
+  // struct textsect {
+  //   uint64_t vaddr;    // Pointer size
+  //   uint64_t length;   // Pointer size
+  //   uint64_t baseaddr; // Pointer size
+  // };
+
+  uint32_t Offset = TextSectAddr + getPsize(); // length field
+  uint64_t RType = Relocation::getAbs(getPsize());
+  uint64_t Addend = -getNewTextStart(BC);
+  BC.addRelocation(Offset, EtextSymbol->getSymbol(), RType, Addend);
+  return 0;
+}
+
+int GolangPass::pcspPass(BinaryFunction *BF, uint8_t **SectionData,
+                         const uint32_t Index, uint8_t Quantum,
+                         bool ForcePreserve) {
+  struct BBVal {
+    BinaryBasicBlock *BB;
+    uint32_t Val;
+    bool IsUncond;
+  } __attribute__((packed)) BBVal = {};
+
+  std::queue<struct BBVal> Queue;
+  std::unordered_map<BinaryBasicBlock *, bool> BBList;
+  InstBias InstBias;
+
+  if (BF->isAsm() || opts::GolangPcspPreserve || ForcePreserve)
+    return writeVarintPass(BF, SectionData, Index, Quantum);
+
+  if (BF->getLayout().block_begin() == BF->getLayout().block_end())
+    return 0;
+
+  BBVal.BB = *BF->getLayout().block_begin();
+  Queue.push(BBVal);
+  while (!Queue.empty()) {
+    BinaryBasicBlock *BB = Queue.front().BB;
+    uint32_t SpVal = Queue.front().Val;
+    bool IsUncond = Queue.front().IsUncond;
+    Queue.pop();
+
+    // We are interested to find condition branching to BB
+    // since uncondition one might be fallthrough
+    auto Search = BBList.find(BB);
+    if (Search != BBList.end() &&
+        (/*not uncond*/ BBList[BB] == false || IsUncond))
+      continue;
+
+    BBList[BB] = IsUncond;
+    BBVal.Val = stackCounter(BF, BB, InstBias, SpVal);
+    for (BinaryBasicBlock *BBS : BB->successors()) {
+      BBVal.BB = BBS;
+      // We use getIndex() here to ensure that originally BBS was right
+      // after BB, so if BB has only one successor no jmp instruction was in BB,
+      // so potentially the BB could have noreturn call
+      BBVal.IsUncond = IsUncond || !!(BB->succ_size() == 1 &&
+                                      BBS->getIndex() == BB->getIndex() + 1);
+      Queue.push(BBVal);
+    }
+  }
+
+  uint64_t Offset, CurrentOffset = 0;
+  int32_t PrevVal = -1, CurrentVal = 0;
+  bool IsFirst = true;
+  if (InstBias.empty())
+    goto out;
+
+  for (auto &I : InstBias) {
+    if (I.second.Val == CurrentVal)
+      continue;
+
+    Offset = I.first;
+    // If the condition was not met it means that the CurrentVal
+    // belongs to the last instruction of the prev BB
+    if (CurrentVal == I.second.OldVal)
+      Offset += I.second.Size;
+
+    writeVarintPair(CurrentVal, PrevVal, Offset, CurrentOffset, IsFirst,
+                    SectionData, Quantum);
+    CurrentVal = I.second.Val;
+  }
+
+  // Add Last Value
+  Offset = InstBias.rbegin()->first + InstBias.rbegin()->second.Size;
+  if (CurrentOffset < Offset) {
+    writeVarintPair(CurrentVal, PrevVal, Offset, CurrentOffset, IsFirst,
+                    SectionData, Quantum);
+  }
+
+out:;
+  **SectionData = 0;
+  (*SectionData)++;
+  return 0;
+}
+
+uint32_t GolangPass::deferreturnPass(BinaryContext &BC,
+                                     BinaryFunction *Function) {
+  for (const BinaryBasicBlock *BB : Function->getLayout().rblocks()) {
+    for (auto II = BB->begin(); II != BB->end(); ++II) {
+      if (!BC.MIB->hasAnnotation(*II, "IsDefer"))
+        continue;
+
+      return getBFInstrOffset(BB, &(*II));
+    }
+  }
+
+  errs() << "BOLT-ERROR: deferreturn call was not found for " << *Function
+         << "\n";
+  exit(1);
+}
+
+int GolangPass::getNextMCinstVal(FunctionLayout::block_iterator BBIt,
+                                 uint64_t I, const uint32_t Index, int32_t &Val,
+                                 uint64_t *NextOffset) {
+  BinaryFunction *BF = (*BBIt)->getFunction();
+  BinaryContext &BC = BF->getBinaryContext();
+  // We're interating in value for the next instruction
+  auto II = std::next((*BBIt)->begin(), I + 1);
+  do {
+    if (II == (*BBIt)->end()) {
+      BBIt = std::next(BBIt);
+      if (BBIt == BF->getLayout().block_end()) {
+        // Last Instruction
+        return -1;
+      }
+
+      II = (*BBIt)->begin();
+    }
+
+    while (II != (*BBIt)->end() && !hasVarintAnnotation(BC, *II, Index)) {
+      if (NextOffset)
+        *NextOffset += BC.computeInstructionSize(*II);
+      II = std::next(II);
+    }
+
+  } while (II == (*BBIt)->end());
+
+  Val = getVarintAnnotation(BC, *II, Index);
+  return 0;
+}
+
+int GolangPass::writeVarintPass(BinaryFunction *BF, uint8_t **DataFuncOffset,
+                                const uint32_t Index, const uint8_t Quantum) {
+  int Ret;
+  uint64_t CurrentOffset = 0, Offset = 0;
+  int32_t PrevVal = -1, Val;
+  bool IsFirst = true;
+  size_t Size = getBFCodeSize(BF);
+  BinaryContext &BC = BF->getBinaryContext();
+  for (auto BBIt = BF->getLayout().block_begin();
+       BBIt != BF->getLayout().block_end(); ++BBIt) {
+    BinaryBasicBlock *BB = *BBIt;
+    for (uint64_t I = 0; I < BB->size(); ++I) {
+      MCInst &Inst = BB->getInstructionAtIndex(I);
+      Offset += BC.computeInstructionSize(Inst);
+      if (Offset < CurrentOffset)
+        continue;
+
+      if (!hasVarintAnnotation(BC, Inst, Index)) {
+        if (Offset == Size && IsFirst)
+          return -1;
+
+        continue;
+      }
+
+      Val = getVarintAnnotation(BC, Inst, Index);
+
+      int32_t NextInstVal;
+      uint64_t NextOffset = Offset;
+      Ret = getNextMCinstVal(BBIt, I, Index, NextInstVal, &NextOffset);
+      if (Ret < 0) {
+        Offset = NextOffset;
+        goto bf_done;
+      }
+
+      if (Val != NextInstVal)
+        writeVarintPair(Val, PrevVal, NextOffset, CurrentOffset, IsFirst,
+                        DataFuncOffset, Quantum);
+    }
+  }
+
+bf_done:;
+  // Create entry for the last instruction
+  writeVarintPair(Val, PrevVal, Offset, CurrentOffset, IsFirst, DataFuncOffset,
+                  Quantum);
+  **DataFuncOffset = 0;
+  (*DataFuncOffset)++;
+  return 0;
+}
+
+static void inlTreePass(BinaryFunction *BF, struct GoFunc *GoFunc,
+                        const char *OldPclntabNames, uint8_t **DataFuncOffset,
+                        const uint8_t *const SectionData) {
+  BinaryContext &BC = BF->getBinaryContext();
+  const unsigned InlIndex = GoFunc->getFuncdataInlTreeIndex();
+  const uint64_t Address = GoFunc->getFuncdata(InlIndex);
+  if (!Address)
+    return;
+
+  static std::unordered_map<uint64_t, uint32_t> InlHash; // String hash, offset
+  ErrorOr<BinarySection &> FuncdataSection = BC.getSectionForAddress(Address);
+  if (!FuncdataSection) {
+    errs() << "BOLT-ERROR: failed to get section for inline 0x"
+           << Twine::utohexstr(Address) << "\n";
+  }
+
+  DataExtractor DE = DataExtractor(FuncdataSection->getContents(),
+                                   BC.AsmInfo->isLittleEndian(),
+                                   BC.AsmInfo->getCodePointerSize());
+
+  std::unordered_map<uint32_t, uint32_t> ParentOffset; // Val:newOffset
+  uint32_t MaxInlCount = 0;
+  for (BinaryBasicBlock *BB : BF->getLayout().blocks()) {
+    for (MCInst &II : *BB) {
+      if (!hasFuncdataAnnotation(BC, II, InlIndex))
+        continue;
+
+      uint32_t Size = getFuncdataSizeAnnotation(BC, II, InlIndex);
+      MaxInlCount += Size;
+      for (uint32_t I = 0; I < Size; ++I) {
+        int32_t Index = getFuncdataAnnotation(BC, II, InlIndex, I);
+        ParentOffset[Index] = getBFInstrOffset(BB, &II);
+      }
+    }
+  }
+
+  for (uint32_t I = 0; I < MaxInlCount; ++I) {
+    uint64_t Offset = Address - FuncdataSection->getAddress();
+    Offset += I * sizeof(InlinedCall);
+
+    struct InlinedCall InlinedCall;
+    uint64_t ReadOffset = Offset;
+    assert(DE.isValidOffset(ReadOffset) && "Invalid offset");
+    InlinedCall.Parent = (int16_t)DE.getU16(&ReadOffset);
+    InlinedCall.FuncID = DE.getU8(&ReadOffset);
+    InlinedCall.Unused = DE.getU8(&ReadOffset);
+    InlinedCall.File = DE.getU32(&ReadOffset);
+    InlinedCall.Line = DE.getU32(&ReadOffset);
+    InlinedCall.Func = DE.getU32(&ReadOffset);
+    InlinedCall.ParentPc = DE.getU32(&ReadOffset);
+
+    // Copy inline function name if it was not copied already
+    const char *Name = OldPclntabNames + InlinedCall.Func;
+    std::hash<std::string> hasher;
+    auto Hash = hasher(std::string(Name));
+    if (InlHash.find(Hash) == InlHash.end()) {
+      InlHash[Hash] = *DataFuncOffset - SectionData;
+      size_t NameLen = strlen(Name) + 1;
+      memcpy(*DataFuncOffset, Name, NameLen);
+      *DataFuncOffset += NameLen;
+    }
+
+    // Use addend as relocation value
+    MCSymbol *ZeroSym = BC.registerNameAtAddress("Zero", 0, 0, 0);
+
+    // Zero out file and line fields
+    FuncdataSection->addRelocation(Offset + offsetof(struct InlinedCall, File),
+                                   ZeroSym,
+                                   Relocation::getAbs(sizeof(uint32_t)));
+    FuncdataSection->addRelocation(Offset + offsetof(struct InlinedCall, Line),
+                                   ZeroSym,
+                                   Relocation::getAbs(sizeof(uint32_t)));
+
+    // Create relocation for inline function name
+    FuncdataSection->addRelocation(
+        Offset + offsetof(struct InlinedCall, Func), ZeroSym,
+        Relocation::getAbs(sizeof(uint32_t)), InlHash[Hash]);
+
+    // Create relocation for parentPc offset
+    FuncdataSection->addRelocation(
+        Offset + offsetof(struct InlinedCall, ParentPc), ZeroSym,
+        Relocation::getAbs(sizeof(uint32_t)), ParentOffset[I]);
+  }
+}
+
+int GolangPass::unsafePointPass(BinaryFunction *BF, GoFunc *GoFunc) {
+  BinaryContext &BC = BF->getBinaryContext();
+  const uint32_t UnsafePointIndex = GoFunc->getPcdataUnsafePointIndex();
+  const int UnsafeVal = GoFunc->getPcdataUnsafePointVal();
+  for (BinaryBasicBlock *BB : BF->getLayout().blocks()) {
+    for (MCInst &Inst : *BB) {
+      bool HasMap = hasVarintAnnotation(BC, Inst, UnsafePointIndex);
+      if (HasMap)
+        continue;
+
+      // The regular branches are the only excpetions for inserted instructions
+      // to be not unsafe point
+      if ((BC.MIB->isBranch(Inst)) && !BC.MIB->isIndirectBranch(Inst))
+        continue;
+
+      addVarintAnnotation(BC, Inst, UnsafePointIndex, UnsafeVal,
+                          /*IsNext*/ false);
+    }
+  }
+
+  return 0;
+}
+
+int GolangPass::pclntabPass(BinaryContext &BC) {
+  int Ret;
+  const uint64_t PclntabAddr = getPcHeaderAddr();
+  if (!PclntabAddr) {
+    errs() << "BOLT-ERROR: pclntab address is zero!\n";
+    return -1;
+  }
+
+  BinaryData *PclntabSym = BC.getBinaryDataAtAddress(PclntabAddr);
+  if (!PclntabSym) {
+    errs() << "BOLT-ERROR: failed to get pclntab symbol!\n";
+    return -1;
+  }
+
+  BinarySection *Section = &PclntabSym->getSection();
+  const unsigned SectionFlags = BinarySection::getFlags(/*IsReadOnly=*/false,
+                                                        /*IsText=*/false,
+                                                        /*IsAllocatable=*/true);
+  uint64_t SectionSize = 0;
+  BinarySection *OutputSection =
+      &BC.registerOrUpdateSection(".pclntab", ELF::SHT_PROGBITS, SectionFlags,
+                                  nullptr, ~0ULL, sizeof(uint64_t));
+
+  // NOTE Currently we don't know how much data we will have in pclntab section.
+  // We would allocate 8 times more then original size.
+  const uint64_t SectionMaxSize =
+      alignTo(PclntabSym->getSize(), BC.RegularPageSize) * 8;
+  uint8_t *const SectionData = new uint8_t[SectionMaxSize];
+  if (!SectionData) {
+    errs() << "BOLT-ERROR: failed to allocate new .pclntab section\n";
+    return -1;
+  }
+
+  DataExtractor DE = DataExtractor(Section->getContents(),
+                                   BC.AsmInfo->isLittleEndian(), getPsize());
+
+  static std::vector<BinaryFunction *> BFs = getSortedGolangFunctions(BC);
+  const uint8_t PclnEntrySize = getPsize() * 2;
+  const size_t BFCount = BFs.size();
+  // Reserve one entry for maxpc, written before FuncPart starts
+  const uint64_t FuncOffset = BFCount * PclnEntrySize + PclnEntrySize;
+  uint8_t *OffsetPart = SectionData + Pclntab->getPcHeaderSize();
+  uint8_t *FuncPart = OffsetPart + FuncOffset;
+
+  // NOTE IsFirstName variable is a hack used due to the bug in go1.16:
+  // https://go-review.googlesource.com/c/go/+/334789
+  bool IsFirstName = true;
+  for (BinaryFunction *BF : BFs) {
+    assert((uint64_t)(FuncPart - SectionData) < SectionMaxSize &&
+           "Overflow error");
+
+    {
+      // Add Functab Offsets
+      uint64_t Delta = (uint64_t)(OffsetPart - SectionData);
+      AddRelaReloc(BC, BF->getSymbol(), OutputSection, Delta);
+      OffsetPart += getPsize();
+      writeEndianPointer(BC, FuncPart - SectionData, &OffsetPart);
+    }
+
+    uint64_t OldTabOffset = BF->getGolangFunctabOffset();
+    std::unique_ptr<struct GoFunc> GoFunc = createGoFunc();
+
+    // Read func structure
+    GoFunc->read(BC, DE, nullptr, &OldTabOffset);
+
+    // Get target structure size
+    size_t FuncSize = GoFunc->getSize(BC);
+    uint8_t *DataFuncOffset = FuncPart + FuncSize;
+
+    // We don't interested in metadata tables anymore
+    GoFunc->disableMetadata();
+
+    // Save space for npcdata
+    DataFuncOffset += GoFunc->getPcdataSize();
+
+    // Save aligned space for nfuncdata
+    DataFuncOffset =
+        SectionData + alignTo(DataFuncOffset - SectionData, getPsize());
+    DataFuncOffset += GoFunc->getNfuncdata() * getPsize();
+
+    // Save name
+    const char *OldPclntabNames =
+        (char *)Section->getData() + Pclntab->getNameOffset();
+    if (GoFunc->getNameOffset() || IsFirstName) {
+      IsFirstName = false;
+      const char *Name = OldPclntabNames + GoFunc->getNameOffset();
+      size_t NameLen = strlen(Name) + 1;
+      memcpy(DataFuncOffset, Name, NameLen);
+      GoFunc->setNameOffset(DataFuncOffset - SectionData);
+      DataFuncOffset += NameLen;
+    }
+
+    // Mark inserted instructions as unsafe points
+    unsafePointPass(BF, GoFunc.get());
+
+    // Fix pcdata
+    auto setPcdata = [&](const uint32_t Index) {
+      GoFunc->setPcdata(Index, DataFuncOffset - SectionData);
+      if (writeVarintPass(BF, &DataFuncOffset, Index, Pclntab->getQuantum()) <
+          0) {
+        GoFunc->setPcdata(Index, 0);
+        return;
+      }
+    };
+
+    setPcdata(GoFunc->getPcdataUnsafePointIndex());
+    setPcdata(GoFunc->getPcdataStackMapIndex());
+    setPcdata(GoFunc->getPcdataInlTreeIndex());
+
+    // Fix npcdata
+    GoFunc->fixNpcdata();
+
+    // Fix funcdata inline
+    inlTreePass(BF, GoFunc.get(), OldPclntabNames, &DataFuncOffset,
+                SectionData);
+
+    // Fix deferreturn
+    if (GoFunc->getDeferreturnOffset())
+      GoFunc->setDeferreturnOffset(deferreturnPass(BC, BF));
+
+    // Fix pcsp
+    if (GoFunc->getPcspOffset()) {
+      // TODO don't preserve PCSP table for functions with deferreturn
+      bool ForcePreserve = GoFunc->getDeferreturnOffset();
+      GoFunc->setPcspOffset(DataFuncOffset - SectionData);
+      Ret = pcspPass(BF, &DataFuncOffset, GoFunc->getPcspIndex(),
+                     Pclntab->getQuantum(), ForcePreserve);
+      if (Ret < 0)
+        goto failed;
+    }
+
+    GoFunc->write(BF, &FuncPart, SectionData, OutputSection);
+    FuncPart = SectionData + alignTo(DataFuncOffset - SectionData, getPsize());
+  }
+
+  SectionSize = FuncPart - SectionData;
+
+  {
+    // The last OffsetPart is maxpc and offset to filetab
+    std::vector<BinaryFunction *> BFs = BC.getSortedFunctions();
+    BinaryFunction *LastBF = getLastBF(BC, BFs);
+    uint64_t Delta = (uint64_t)(OffsetPart - SectionData);
+    AddRelaReloc(BC, LastBF->getSymbol(), OutputSection, Delta);
+    OffsetPart += getPsize();
+    writeEndianPointer(BC, 0, &OffsetPart);
+  }
+
+  // Write fixed Pclntab structure
+  Pclntab->setFunctionsCount(BFCount);
+  Pclntab->writeHeader(BC, SectionData);
+
+  // Fix section sizes size
+  FirstModule->setPclntabSize(SectionSize);
+
+  // Fix ftab size
+  FirstModule->setFtabSize(BFCount);
+
+  OutputSection->updateContents(SectionData, SectionSize);
+  OutputSection->setIsFinalized();
+
+  PclntabSym->setOutputSize(SectionSize);
+  PclntabSym->setOutputLocation(*OutputSection, 0);
+  return 0;
+
+failed:;
+  delete[] SectionData;
+  return Ret;
+}
+
+int GolangPass::findFuncTabPass(BinaryContext &BC) {
+  // const uint32_t minfunc = 16; // runtime/symtab.go: minimum function size
+  const uint32_t pcsubbucketssize = 256;
+  // const uint32_t pcbucketsize = pcsubbucketssize * minfunc; //
+  // runtime/symtab.go: size of bucket in the pc->func lookup table
+
+  uint64_t FindFuncTab = FirstModule->getFindfunctab();
+  if (!FindFuncTab) {
+    errs() << "BOLT-ERROR: findfunctab is zero!\n";
+    return -1;
+  }
+
+  BinaryData *FindfunctabSym = BC.getBinaryDataAtAddress(FindFuncTab);
+  if (!FindfunctabSym) {
+    errs() << "BOLT-ERROR: failed to get findfunctab symbol!\n";
+    return -1;
+  }
+
+  const unsigned SectionFlags = BinarySection::getFlags(/*IsReadOnly=*/true,
+                                                        /*IsText=*/false,
+                                                        /*IsAllocatable=*/true);
+  uint64_t SectionSize = 0;
+  BinarySection *OutputSection = &BC.registerOrUpdateSection(
+      ".findfunctab", ELF::SHT_PROGBITS, SectionFlags, nullptr, ~0ULL,
+      sizeof(uint64_t));
+  // runtime/symtab.go
+  struct {
+    uint32_t Idx;
+    uint8_t Subbuckets[16];
+  } Findfuncbucket;
+
+  // NOTE Currently we don't know how much BFs occupy in text section.
+  // We will allocate 4 times more then original size using mmap.
+  const uint64_t SectionMaxSize =
+      alignTo(FindfunctabSym->getSize(), BC.RegularPageSize) * 4;
+  uint8_t *const SectionData = new uint8_t[SectionMaxSize];
+  if (!SectionData) {
+    errs() << "BOLT-ERROR: failed to allocate new .findfunctab section\n";
+    return -1;
+  }
+
+  uint32_t LastIdx = 0, SubIndex = 0, Index = 0;
+  uint64_t Offset = getNewTextStart(BC),
+           NextOffset = Offset; // NOTE For align calc
+  uint8_t *Data = SectionData;
+  std::vector<BinaryFunction *> BFs = getSortedGolangFunctions(BC);
+  for (auto BFit = BFs.begin(); BFit != BFs.end(); ++BFit) {
+    assert((uint64_t)(Data - SectionData) < SectionMaxSize && "Overflow error");
+    BinaryFunction *BF = *BFit;
+    uint64_t Size = getBFSize(BF);
+    Offset += Size;
+    auto BFNext = std::next(BFit, 1);
+    if (BFNext != BFs.end()) {
+      if (BC.HasRelocations)
+        Offset = alignTo(Offset, BinaryFunction::MinAlign);
+
+      unsigned Alignment, MaxAlignment;
+      std::tie(Alignment, MaxAlignment) =
+          BC.getBFAlignment(**BFNext, /*EmitColdPart*/ false);
+      uint64_t Pad = offsetToAlignment(Offset, llvm::Align(Alignment));
+      if (Pad <= MaxAlignment)
+        Offset += Pad;
+
+      // Cold part start, align section
+      if (BF->getIndex() < INVALID_BF_INDEX &&
+          (*BFNext)->getIndex() == INVALID_BF_INDEX)
+        Offset = alignTo(Offset, opts::AlignFunctions);
+    }
+
+    // Offset points to the next BF
+    // NextOffset points to the pcsubbucketssize aligned address somewhere
+    // in the current BF
+    if (Offset <= NextOffset) {
+      ++Index;
+      continue;
+    }
+
+    // We are interested in the part of the function starting from NextOffset
+    Size = Offset - NextOffset;
+    do {
+      if (SubIndex % sizeof(Findfuncbucket.Subbuckets) == 0) {
+        writeEndianVal(BC, Index, sizeof(Findfuncbucket.Idx), &Data);
+        LastIdx = Index;
+        SubIndex = 0;
+      }
+
+      *Data++ = Index - LastIdx;
+      ++SubIndex;
+      Size -= pcsubbucketssize;
+    } while ((int64_t)Size > 0);
+
+    NextOffset = alignTo(Offset, pcsubbucketssize);
+    ++Index;
+  }
+
+  SectionSize = Data - SectionData;
+  OutputSection->updateContents(SectionData, SectionSize);
+  FindfunctabSym->setOutputSize(SectionSize);
+  FindfunctabSym->setOutputLocation(*OutputSection, 0);
+
+  // NOTE To be able to emit new data we need to have at least one relocation
+  // for OutputSection to be created. Since the first 4 bytes of the findfunctab
+  // is always 0 create dummy 4 bytes abs relocation there
+  MCSymbol *ZeroSym = BC.registerNameAtAddress("Zero", 0, 0, 0);
+  OutputSection->addRelocation(0, ZeroSym,
+                               Relocation::getAbs(sizeof(uint32_t)));
+  OutputSection->setIsFinalized();
+  return 0;
+}
+
+int GolangPass::getSymbols(BinaryContext &BC) {
+  // The iface/eface ifn/tfn addresses are set relative to this symbol
+  BinaryData *TextSymbol = BC.getFirstBinaryDataByName(getFirstBFName());
+  if (!TextSymbol) {
+    outs() << "BOLT-WARNING: failed to get text start symbol!\n";
+    return -1;
+  }
+
+  RuntimeText = TextSymbol->getAddress();
+  return 0;
+}
+
+int GolangPass::checkGoVersion(BinaryContext &BC) {
+  auto failed = [&](void) -> int {
+    outs() << "BOLT-WARNING: could not idetifiy Go version for input binary!\n";
+    if (opts::GolangPass != opts::GV_AUTO)
+      return 0;
+
+    outs() << "BOLT-ERROR: no compatible version found! Specify gc version "
+              "explicitly\n";
+    return -1;
+  };
+
+  BinaryData *BuildVersion =
+      BC.getFirstBinaryDataByName("runtime.buildVersion");
+  if (!BuildVersion)
+    return failed();
+
+  BinarySection *Section = &BuildVersion->getSection();
+  DataExtractor DE =
+      DataExtractor(Section->getContents(), BC.AsmInfo->isLittleEndian(),
+                    BC.AsmInfo->getCodePointerSize());
+
+  uint64_t GoVersionOffset = BuildVersion->getAddress() - Section->getAddress();
+  uint64_t GoVersionAddr = DE.getAddress(&GoVersionOffset);
+  ErrorOr<BinarySection &> GoVersionSection =
+      BC.getSectionForAddress(GoVersionAddr);
+  if (!GoVersionSection) {
+    errs()
+        << "BOLT-ERROR: failed to get binary section for go version string\n";
+    return failed();
+  }
+
+  const char *BinaryVersion = GoVersionSection->getContents().data();
+  BinaryVersion += (GoVersionAddr - GoVersionSection->getAddress());
+
+  const unsigned MaxVerLen = 9;
+  if (opts::GolangPass != opts::GV_AUTO) {
+    const char *ExpectedVersion = GolangStringVer[opts::GolangPass];
+    if (memcmp(BinaryVersion, ExpectedVersion, strlen(ExpectedVersion))) {
+      // NOTE Limit expected version string to 9 chars
+      outs() << "BOLT-WARNING: the binary expected version is "
+             << ExpectedVersion
+             << " but found: " << std::string(BinaryVersion, MaxVerLen) << "\n";
+      return 0;
+    }
+  } else {
+    for (int I = opts::GV_LATEST; I > opts::GV_FIRST; --I) {
+      if (!memcmp(BinaryVersion, GolangStringVer[I],
+                  strlen(GolangStringVer[I]))) {
+        outs() << "BOLT-INFO: golang version is: " << GolangStringVer[I]
+               << "\n";
+        opts::GolangPass = (opts::GolangVersion)I;
+        return 0;
+      }
+    }
+
+    outs() << "BOLT-INFO: the binary version is: "
+           << std::string(BinaryVersion, MaxVerLen) << "\n";
+    return failed();
+  }
+
+  return 0;
+}
+
+void GolangPass::runOnFunctions(BinaryContext &BC) {
+  int Ret;
+
+#define CALL_STAGE(func)                                                       \
+  Ret = func(BC);                                                              \
+  if (Ret < 0) {                                                               \
+    errs() << "BOLT-ERROR: golang " << #func << " stage failed!\n";            \
+    exit(1);                                                                   \
+  }
+
+  CALL_STAGE(updateBBSizes);
+
+  CALL_STAGE(typelinksPass);
+
+  CALL_STAGE(pclntabPass);
+
+  CALL_STAGE(findFuncTabPass);
+
+  CALL_STAGE(textsectmapPass);
+
+  CALL_STAGE(FirstModule->patch);
+
+#undef CALL_STAGE
+}
+
+} // end namespace bolt
+} // end namespace llvm

bolt/lib/Passes/IdenticalCodeFolding.cpp

 //===- bolt/Passes/IdenticalCodeFolding.cpp -------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the IdenticalCodeFolding class.
 //
 //===----------------------------------------------------------------------===//
 
 #include "bolt/Passes/IdenticalCodeFolding.h"
 #include "bolt/Core/ParallelUtilities.h"
+#include "bolt/Passes/Golang.h"
+#include "bolt/Utils/CommandLineOpts.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ThreadPool.h"
 #include "llvm/Support/Timer.h"
 #include <atomic>
 #include <iterator>
 #include <map>
 #include <set>
if (EHInfoA && EHInfoB) {
         assert(LPA && LPB && "cannot locate landing pad(s)");
 
         if (LPA->getLayoutIndex() != LPB->getLayoutIndex())
           return false;
       }
     }
   }
 
+  if (opts::GolangPass != opts::GV_NONE &&
+      !BC.MIB->areAnnotationsEqual(InstA, InstB, /*GenericOnly*/ true))
+    return false;
+
   return BC.MIB->equals(InstA, InstB, Comp);
 }
 
 /// Returns true if this function has identical code and CFG with
 /// the given function \p BF.
 ///
 /// If \p CongruentSymbols is set to true, then symbolic operands that reference
 /// potentially identical but different functions are ignored during the
static bool isIdenticalWith(const BinaryFunction &A, const BinaryFunction &B,
 
   // Comparing multi-entry functions could be non-trivial.
   if (A.isMultiEntry() || B.isMultiEntry())
     return false;
 
   if (A.hasIslandsInfo() || B.hasIslandsInfo())
     return false;
 
+  if (opts::GolangPass != opts::GV_NONE) {
+    std::unique_ptr<struct GoFunc> GoFuncA = createGoFunc();
+    std::unique_ptr<struct GoFunc> GoFuncB = createGoFunc();
+    GoFuncA->read(A);
+    GoFuncB->read(B);
+
+    if (GoFuncA->getNpcdata() != GoFuncB->getNpcdata())
+      return false;
+
+    if (GoFuncA->getNfuncdata() != GoFuncB->getNfuncdata())
+      return false;
+
+    for (unsigned I = 0; I < GoFuncA->getNfuncdata(); ++I)
+      if (GoFuncA->getFuncdata(I) != GoFuncB->getFuncdata(I))
+        return false;
+  }
+
   // Process both functions in either DFS or existing order.
   SmallVector<const BinaryBasicBlock *, 0> OrderA;
   SmallVector<const BinaryBasicBlock *, 0> OrderB;
   if (opts::UseDFS) {
     copy(A.dfs(), std::back_inserter(OrderA));
     copy(B.dfs(), std::back_inserter(OrderB));
   } else {
     copy(A.getLayout().blocks(), std::back_inserter(OrderA));
if (Operand.isExpr())
     return hashExpr(BC, *Operand.getExpr());
 
   return std::string();
 }
 
 namespace llvm {
 namespace bolt {
 
+bool IdenticalCodeFolding::shouldOptimizeICF(const BinaryFunction &BF) const {
+  if (BF.hasUnknownControlFlow())
+    return false;
+  if (BF.isFolded())
+    return false;
+  if (BF.hasSDTMarker())
+    return false;
+
+  if (opts::GolangPass != opts::GV_NONE) {
+    if (!BF.isGolang())
+      return false;
+
+    if (BF.getIndex() == GO_FIRST_BF_INDEX || BF.getIndex() == GO_LAST_BF_INDEX)
+      return false;
+  }
+
+  return BinaryFunctionPass::shouldOptimize(BF);
+}
+
 void IdenticalCodeFolding::runOnFunctions(BinaryContext &BC) {
   const size_t OriginalFunctionCount = BC.getBinaryFunctions().size();
   uint64_t NumFunctionsFolded = 0;
   std::atomic<uint64_t> NumJTFunctionsFolded{0};
   std::atomic<uint64_t> BytesSavedEstimate{0};
   std::atomic<uint64_t> NumCalled{0};
   std::atomic<uint64_t> NumFoldedLastIteration{0};
   CongruentBucketsMap CongruentBuckets;
void IdenticalCodeFolding::runOnFunctions(BinaryContext &BC) {
   // Creates buckets with congruent functions - functions that potentially
   // could  be folded.
   auto createCongruentBuckets = [&]() {
     NamedRegionTimer CongruentBucketsTimer("congruent buckets",
                                            "congruent buckets", "ICF breakdown",
                                            "ICF breakdown", opts::TimeICF);
     for (auto &BFI : BC.getBinaryFunctions()) {
       BinaryFunction &BF = BFI.second;
-      if (!this->shouldOptimize(BF))
+      if (!shouldOptimizeICF(BF))
         continue;
       CongruentBuckets[&BF].emplace(&BF);
     }
   };
 
   // Partition each set of congruent functions into sets of identical functions
   // and fold them
   auto performFoldingPass = [&]() {

bolt/lib/Passes/Inliner.cpp

static cl::opt<bool> InlineSmallFunctions(
     cl::cat(BoltOptCategory));
 
 static cl::opt<unsigned> InlineSmallFunctionsBytes(
     "inline-small-functions-bytes",
     cl::desc("max number of bytes for the function to be considered small for "
              "inlining purposes"),
     cl::init(4), cl::Hidden, cl::cat(BoltOptCategory));
 
-static cl::opt<bool> NoInline(
+cl::opt<bool> NoInline(
     "no-inline",
     cl::desc("disable all inlining (overrides other inlining options)"),
-    cl::cat(BoltOptCategory));
+    cl::init(false), cl::ZeroOrMore, cl::cat(BoltOptCategory));
 
 /// This function returns true if any of inlining options are specified and the
 /// inlining pass should be executed. Whenever a new inlining option is added,
 /// this function should reflect the change.
 bool inliningEnabled() {
   return !NoInline &&
          (InlineAll || InlineSmallFunctions || !ForceInlineFunctions.empty());
 }

bolt/lib/Passes/Instrumentation.cpp

Instrumentation::createInstrumentationSnippet(BinaryContext &BC, bool IsLeaf) {
   Summary->Counters.emplace_back(Label);
   return BC.MIB->createInstrIncMemory(Label, BC.Ctx.get(), IsLeaf);
 }
 
 // Helper instruction sequence insertion function
 static BinaryBasicBlock::iterator
 insertInstructions(InstructionListType &Instrs, BinaryBasicBlock &BB,
                    BinaryBasicBlock::iterator Iter) {
+  BinaryContext &BC = BB.getFunction()->getBinaryContext();
   for (MCInst &NewInst : Instrs) {
+    BC.MIB->getOrCreateAnnotationAs<bool>(NewInst, "IsInstrumentation") = true;
     Iter = BB.insertInstruction(Iter, NewInst);
     ++Iter;
   }
   return Iter;
 }
 
 void Instrumentation::instrumentLeafNode(BinaryBasicBlock &BB,
                                          BinaryBasicBlock::iterator Iter,
for (const MCInst &Inst : BB) {
         if (BC.MIB->isInvoke(Inst))
           InvokeBlocks.insert(&BB);
         if (!BC.MIB->isTailCall(Inst))
           IsLeafFunction = false;
       }
     }
   }
 
+  // If we have a suspicion that the function was written in ASM
+  // lets treat it as a leaf function since the code there might
+  // use stack lower than a function frame and we can wipe saved
+  // values on stack by instrumenting it
+  // TODO Track SP access by the function to determine such cases
+  if (Function.isAsm())
+    IsLeafFunction = true;
+
   for (auto BBI = Function.begin(), BBE = Function.end(); BBI != BBE; ++BBI) {
     BinaryBasicBlock &BB = *BBI;
     bool HasUnconditionalBranch = false;
     bool HasJumpTable = false;
     bool IsInvokeBlock = InvokeBlocks.count(&BB) > 0;
 
     for (auto I = BB.begin(); I != BB.end(); ++I) {
       const MCInst &Inst = *I;
void Instrumentation::instrumentFunction(BinaryFunction &Function,
   // Unused now
   FuncDesc->EdgesSet.clear();
 }
 
 void Instrumentation::runOnFunctions(BinaryContext &BC) {
   if (!BC.isX86())
     return;
 
+  BC.MIB->getOrCreateAnnotationIndex("IsInstrumentation");
+
   const unsigned Flags = BinarySection::getFlags(/*IsReadOnly=*/false,
                                                  /*IsText=*/false,
                                                  /*IsAllocatable=*/true);
   BC.registerOrUpdateSection(".bolt.instr.counters", ELF::SHT_PROGBITS, Flags,
                              nullptr, 0, 1);
 
   BC.registerOrUpdateNoteSection(".bolt.instr.tables", nullptr, 0,
                                  /*Alignment=*/1,

bolt/lib/Passes/LongJmp.cpp

static cl::opt<bool> GroupStubs("group-stubs",
                                 cl::init(true), cl::cat(BoltOptCategory));
 }
 
 namespace llvm {
 namespace bolt {
 
 constexpr unsigned ColdFragAlign = 16;
 
+inline uint64_t getInstructionSize(const BinaryContext &BC,
+                                   const MCInst &Inst) {
+  if (BC.isAArch64())
+    return 4;
+  return BC.computeInstructionSize(Inst);
+}
+
 static void relaxStubToShortJmp(BinaryBasicBlock &StubBB, const MCSymbol *Tgt) {
+
   const BinaryContext &BC = StubBB.getFunction()->getBinaryContext();
   InstructionListType Seq;
   BC.MIB->createShortJmp(Seq, Tgt, BC.Ctx.get());
   StubBB.clear();
   StubBB.addInstructions(Seq.begin(), Seq.end());
 }
 
 static void relaxStubToLongJmp(BinaryBasicBlock &StubBB, const MCSymbol *Tgt) {
for (auto I = Func.getLayout().block_begin(),
     auto Next = std::next(I);
     if (Next != E && (*Next)->isCold())
       return *I;
   }
   llvm_unreachable("No hot-colt split point found");
 }
 
 static bool shouldInsertStub(const BinaryContext &BC, const MCInst &Inst) {
+  // We are not insterested in calls relaxation for X86. Also it has a problem
+  // with inserted functions e.g. instrumentation indirect calls since we
+  // did not update layout for them
+  if (BC.isX86())
+    return BC.MIB->isBranch(Inst) && !BC.MIB->isIndirectBranch(Inst);
+
   return (BC.MIB->isBranch(Inst) || BC.MIB->isCall(Inst)) &&
          !BC.MIB->isIndirectBranch(Inst) && !BC.MIB->isIndirectCall(Inst);
 }
 
 std::pair<std::unique_ptr<BinaryBasicBlock>, MCSymbol *>
 LongJmpPass::createNewStub(BinaryBasicBlock &SourceBB, const MCSymbol *TgtSym,
                            bool TgtIsFunc, uint64_t AtAddress) {
   BinaryFunction &Func = *SourceBB.getFunction();
if (Cand == Candidates.end())
     return nullptr;
   if (Cand != Candidates.begin()) {
     const StubTy *LeftCand = std::prev(Cand);
     if (Cand->first - DotAddress > DotAddress - LeftCand->first)
       Cand = LeftCand;
   }
   int BitsAvail = BC.MIB->getPCRelEncodingSize(Inst) - 1;
   uint64_t Mask = ~((1ULL << BitsAvail) - 1);
-  uint64_t PCRelTgtAddress = Cand->first;
-  PCRelTgtAddress = DotAddress > PCRelTgtAddress ? DotAddress - PCRelTgtAddress
-                                                 : PCRelTgtAddress - DotAddress;
+  uint64_t InstSize = getInstructionSize(BC, Inst);
+  uint64_t PCRelTgtAddress =
+      getTargetOffset(BC, InstSize, Cand->first, DotAddress);
   LLVM_DEBUG({
     if (Candidates.size() > 1)
       dbgs() << "Considering stub group with " << Candidates.size()
              << " candidates. DotAddress is " << Twine::utohexstr(DotAddress)
              << ", chosen candidate address is "
              << Twine::utohexstr(Cand->first) << "\n";
   });
   return PCRelTgtAddress & Mask ? nullptr : Cand->second;
for (auto &KeyVal : HotLocalStubs)
     update(KeyVal.second);
   for (auto &KeyVal : ColdLocalStubs)
     update(KeyVal.second);
   update(HotStubGroups);
   update(ColdStubGroups);
 }
 
 void LongJmpPass::tentativeBBLayout(const BinaryFunction &Func) {
-  const BinaryContext &BC = Func.getBinaryContext();
   uint64_t HotDot = HotAddresses[&Func];
   uint64_t ColdDot = ColdAddresses[&Func];
   bool Cold = false;
   for (const BinaryBasicBlock *BB : Func.getLayout().blocks()) {
     if (Cold || BB->isCold()) {
       Cold = true;
       BBAddresses[BB] = ColdDot;
-      ColdDot += BC.computeCodeSize(BB->begin(), BB->end());
+      ColdDot += BB->estimateSize();
     } else {
       BBAddresses[BB] = HotDot;
-      HotDot += BC.computeCodeSize(BB->begin(), BB->end());
+      HotDot += BB->estimateSize();
     }
   }
 }
 
 uint64_t LongJmpPass::tentativeLayoutRelocColdPart(
     const BinaryContext &BC, std::vector<BinaryFunction *> &SortedFunctions,
     uint64_t DotAddress) {
   DotAddress = alignTo(DotAddress, llvm::Align(opts::AlignFunctions));
   for (BinaryFunction *Func : SortedFunctions) {
     if (!Func->isSplit())
       continue;
     DotAddress = alignTo(DotAddress, BinaryFunction::MinAlign);
-    uint64_t Pad =
-        offsetToAlignment(DotAddress, llvm::Align(Func->getAlignment()));
-    if (Pad <= Func->getMaxColdAlignmentBytes())
+    unsigned Alignment, MaxAlignment;
+    std::tie(Alignment, MaxAlignment) =
+        BC.getBFAlignment(*Func, /*EmitColdPart*/ true);
+    uint64_t Pad = offsetToAlignment(DotAddress, llvm::Align(Alignment));
+    if (Pad <= MaxAlignment)
       DotAddress += Pad;
     ColdAddresses[Func] = DotAddress;
     LLVM_DEBUG(dbgs() << Func->getPrintName() << " cold tentative: "
                       << Twine::utohexstr(DotAddress) << "\n");
     DotAddress += Func->estimateColdSize();
     DotAddress = alignTo(DotAddress, Func->getConstantIslandAlignment());
     DotAddress += Func->estimateConstantIslandSize();
   }
for (BinaryFunction *Func : SortedFunctions) {
 
     if (!ColdLayoutDone && CurrentIndex >= LastHotIndex) {
       DotAddress =
           tentativeLayoutRelocColdPart(BC, SortedFunctions, DotAddress);
       ColdLayoutDone = true;
       if (opts::HotFunctionsAtEnd)
         DotAddress = alignTo(DotAddress, opts::AlignText);
     }
-
     DotAddress = alignTo(DotAddress, BinaryFunction::MinAlign);
-    uint64_t Pad =
-        offsetToAlignment(DotAddress, llvm::Align(Func->getAlignment()));
-    if (Pad <= Func->getMaxAlignmentBytes())
+    unsigned Alignment, MaxAlignment;
+    std::tie(Alignment, MaxAlignment) =
+        BC.getBFAlignment(*Func, /*EmitColdPart*/ false);
+    uint64_t Pad = offsetToAlignment(DotAddress, llvm::Align(Alignment));
+    if (Pad <= MaxAlignment)
       DotAddress += Pad;
     HotAddresses[Func] = DotAddress;
     LLVM_DEBUG(dbgs() << Func->getPrintName() << " tentative: "
                       << Twine::utohexstr(DotAddress) << "\n");
     if (!Func->isSplit())
       DotAddress += Func->estimateSize();
     else
       DotAddress += Func->estimateHotSize();
if (Iter == HotAddresses.end() || (TargetFunc && EntryID)) {
     return *ValueOrError;
   }
   return Iter->second;
 }
 
 bool LongJmpPass::relaxStub(BinaryBasicBlock &StubBB) {
   const BinaryFunction &Func = *StubBB.getFunction();
   const BinaryContext &BC = Func.getBinaryContext();
+  assert(BC.isAArch64() && "Unsupported arch");
   const int Bits = StubBits[&StubBB];
   // Already working with the largest range?
   if (Bits == static_cast<int>(BC.AsmInfo->getCodePointerSize() * 8))
     return false;
 
   const static int RangeShortJmp = BC.MIB->getShortJmpEncodingSize();
   const static int RangeSingleInstr = BC.MIB->getUncondBranchEncodingSize();
   const static uint64_t ShortJmpMask = ~((1ULL << RangeShortJmp) - 1);
   const static uint64_t SingleInstrMask =
       ~((1ULL << (RangeSingleInstr - 1)) - 1);
 
   const MCSymbol *RealTargetSym = BC.MIB->getTargetSymbol(*StubBB.begin());
-  const BinaryBasicBlock *TgtBB = Func.getBasicBlockForLabel(RealTargetSym);
-  uint64_t TgtAddress = getSymbolAddress(BC, RealTargetSym, TgtBB);
   uint64_t DotAddress = BBAddresses[&StubBB];
-  uint64_t PCRelTgtAddress = DotAddress > TgtAddress ? DotAddress - TgtAddress
-                                                     : TgtAddress - DotAddress;
+  const uint64_t InstSize = 4;
+  uint64_t PCRelTgtAddress =
+      getTargetOffset(Func, InstSize, RealTargetSym, DotAddress);
+
   // If it fits in one instruction, do not relax
   if (!(PCRelTgtAddress & SingleInstrMask))
     return false;
 
   // Fits short jmp
   if (!(PCRelTgtAddress & ShortJmpMask)) {
     if (Bits >= RangeShortJmp)
       return false;
bool LongJmpPass::relaxStub(BinaryBasicBlock &StubBB) {
   LLVM_DEBUG(dbgs() << "Relaxing stub to long jump. PCRelTgtAddress = "
                     << Twine::utohexstr(PCRelTgtAddress)
                     << " RealTargetSym = " << RealTargetSym->getName() << "\n");
   relaxStubToLongJmp(StubBB, RealTargetSym);
   StubBits[&StubBB] = static_cast<int>(BC.AsmInfo->getCodePointerSize() * 8);
   return true;
 }
 
-bool LongJmpPass::needsStub(const BinaryBasicBlock &BB, const MCInst &Inst,
+uint64_t LongJmpPass::getTargetOffset(const BinaryContext &BC,
+                                      uint64_t InstSize, uint64_t TargetAddress,
                                       uint64_t DotAddress) const {
-  const BinaryFunction &Func = *BB.getFunction();
-  const BinaryContext &BC = Func.getBinaryContext();
-  const MCSymbol *TgtSym = BC.MIB->getTargetSymbol(Inst);
-  assert(TgtSym && "getTargetSymbol failed");
+  uint64_t PCRelTgtAddress;
+  PCRelTgtAddress = DotAddress > TargetAddress ? DotAddress - TargetAddress
+                                               : TargetAddress - DotAddress;
+  if (BC.isX86() && TargetAddress < DotAddress) {
+    // For x86 we need to take into account instruction size
+    PCRelTgtAddress += InstSize;
+  }
 
+  return PCRelTgtAddress;
+}
+
+uint64_t LongJmpPass::getTargetOffset(const BinaryFunction &Func,
+                                      uint64_t InstSize, const MCSymbol *TgtSym,
+                                      uint64_t DotAddress) const {
+  const BinaryContext &BC = Func.getBinaryContext();
   const BinaryBasicBlock *TgtBB = Func.getBasicBlockForLabel(TgtSym);
   // Check for shared stubs from foreign functions
   if (!TgtBB) {
     auto SSIter = SharedStubs.find(TgtSym);
-    if (SSIter != SharedStubs.end())
+    if (SSIter != SharedStubs.end()) {
       TgtBB = SSIter->second;
     }
+  }
 
-  int BitsAvail = BC.MIB->getPCRelEncodingSize(Inst) - 1;
-  uint64_t Mask = ~((1ULL << BitsAvail) - 1);
+  uint64_t Address = getSymbolAddress(BC, TgtSym, TgtBB);
+  return getTargetOffset(BC, InstSize, Address, DotAddress);
+}
 
-  uint64_t PCRelTgtAddress = getSymbolAddress(BC, TgtSym, TgtBB);
-  PCRelTgtAddress = DotAddress > PCRelTgtAddress ? DotAddress - PCRelTgtAddress
-                                                 : PCRelTgtAddress - DotAddress;
+uint64_t LongJmpPass::getTargetOffset(const BinaryFunction &Func,
+                                      const MCInst &Inst,
+                                      uint64_t DotAddress) const {
+  const BinaryContext &BC = Func.getBinaryContext();
+  const MCSymbol *TgtSym = BC.MIB->getTargetSymbol(Inst);
+  assert(TgtSym && "getTargetSymbol failed");
+  uint64_t InstSize = getInstructionSize(BC, Inst);
+  return getTargetOffset(Func, InstSize, TgtSym, DotAddress);
+}
 
-  return PCRelTgtAddress & Mask;
+bool LongJmpPass::needsStub(const BinaryBasicBlock &BB, const MCInst &Inst,
+                            uint64_t DotAddress) const {
+  const BinaryFunction &Func = *BB.getFunction();
+  const BinaryContext &BC = Func.getBinaryContext();
+  uint64_t PCRelTgtAddress = getTargetOffset(Func, Inst, DotAddress);
+  int BitsAvail = BC.MIB->getPCRelEncodingSize(Inst) - 1;
+  uint64_t Mask = ~((1ULL << BitsAvail) - 1);
+  return !!(PCRelTgtAddress & Mask);
 }
 
-bool LongJmpPass::relax(BinaryFunction &Func) {
+LongJmpPass::RelaxRet LongJmpPass::relax(BinaryFunction &Func) {
   const BinaryContext &BC = Func.getBinaryContext();
-  bool Modified = false;
+  RelaxRet Modified = RelaxRet::NotModified;
 
-  assert(BC.isAArch64() && "Unsupported arch");
-  constexpr int InsnSize = 4; // AArch64
   std::vector<std::pair<BinaryBasicBlock *, std::unique_ptr<BinaryBasicBlock>>>
       Insertions;
 
   BinaryBasicBlock *Frontier = getBBAtHotColdSplitPoint(Func);
   uint64_t FrontierAddress = Frontier ? BBAddresses[Frontier] : 0;
   if (FrontierAddress)
-    FrontierAddress += Frontier->getNumNonPseudos() * InsnSize;
+    FrontierAddress += Frontier->estimateSize();
 
   // Add necessary stubs for branch targets we know we can't fit in the
   // instruction
   for (BinaryBasicBlock &BB : Func) {
     uint64_t DotAddress = BBAddresses[&BB];
     // Stubs themselves are relaxed on the next loop
     if (Stubs[&Func].count(&BB))
       continue;
 
-    for (MCInst &Inst : BB) {
+    for (auto II = BB.begin(); II != BB.end(); ++II) {
+      MCInst &Inst = *II;
       if (BC.MIB->isPseudo(Inst))
         continue;
 
+      uint64_t InstSize = getInstructionSize(BC, Inst);
       if (!shouldInsertStub(BC, Inst)) {
-        DotAddress += InsnSize;
+        DotAddress += InstSize;
         continue;
       }
 
       // Check and relax direct branch or call
       if (!needsStub(BB, Inst, DotAddress)) {
-        DotAddress += InsnSize;
+        DotAddress += InstSize;
+        continue;
+      }
+
+      Modified =
+          static_cast<RelaxRet>(static_cast<int>(Modified) |
+                                static_cast<int>(RelaxRet::InstrRelaxed));
+
+      // Try to relax instruction in-place, we will check if it fits
+      // on the next round
+      if (BC.MIB->relaxInstruction(Inst)) {
+        DotAddress += getInstructionSize(BC, Inst);
         continue;
       }
-      Modified = true;
 
       // Insert stubs close to the patched BB if call, but far away from the
       // hot path if a branch, since this branch target is the cold region
       // (but first check that the far away stub will be in range).
       BinaryBasicBlock *InsertionPoint = &BB;
       if (Func.isSimple() && !BC.MIB->isCall(Inst) && FrontierAddress &&
           !BB.isCold()) {
         int BitsAvail = BC.MIB->getPCRelEncodingSize(Inst) - 1;
         uint64_t Mask = ~((1ULL << BitsAvail) - 1);
         assert(FrontierAddress > DotAddress &&
                "Hot code should be before the frontier");
-        uint64_t PCRelTgt = FrontierAddress - DotAddress;
+        uint64_t PCRelTgt =
+            getTargetOffset(BC, InstSize, FrontierAddress, DotAddress);
         if (!(PCRelTgt & Mask))
           InsertionPoint = Frontier;
       }
+
       // Always put stubs at the end of the function if non-simple. We can't
       // change the layout of non-simple functions because it has jump tables
       // that we do not control.
       if (!Func.isSimple())
         InsertionPoint = &*std::prev(Func.end());
 
+      Modified =
+          static_cast<RelaxRet>(static_cast<int>(Modified) |
+                                static_cast<int>(RelaxRet::StubsInserted));
+
       // Create a stub to handle a far-away target
       Insertions.emplace_back(InsertionPoint,
                               replaceTargetWithStub(BB, Inst, DotAddress,
                                                     InsertionPoint == Frontier
                                                         ? FrontierAddress
                                                         : DotAddress));
 
-      DotAddress += InsnSize;
+      DotAddress += InstSize;
     }
   }
 
   // Relax stubs if necessary
   for (BinaryBasicBlock &BB : Func) {
     if (!Stubs[&Func].count(&BB) || !BB.isValid())
       continue;
 
-    Modified |= relaxStub(BB);
+    if (relaxStub(BB))
+      Modified =
+          static_cast<RelaxRet>(static_cast<int>(Modified) |
+                                static_cast<int>(RelaxRet::InstrRelaxed));
   }
 
   for (std::pair<BinaryBasicBlock *, std::unique_ptr<BinaryBasicBlock>> &Elmt :
        Insertions) {
     if (!Elmt.second)
       continue;
     std::vector<std::unique_ptr<BinaryBasicBlock>> NewBBs;
     NewBBs.emplace_back(std::move(Elmt.second));
     Func.insertBasicBlocks(Elmt.first, std::move(NewBBs), true);
   }
 
   return Modified;
 }
 
 void LongJmpPass::runOnFunctions(BinaryContext &BC) {
   outs() << "BOLT-INFO: Starting stub-insertion pass\n";
   std::vector<BinaryFunction *> Sorted = BC.getSortedFunctions();
   bool Modified;
   uint32_t Iterations = 0;
   do {
-    ++Iterations;
     Modified = false;
+    ++Iterations;
     tentativeLayout(BC, Sorted);
     updateStubGroups();
     for (BinaryFunction *Func : Sorted) {
-      if (relax(*Func)) {
-        // Don't ruin non-simple functions, they can't afford to have the layout
-        // changed.
-        if (Func->isSimple())
-          Func->fixBranches();
-        Modified = true;
-      }
+      RelaxRet Ret = relax(*Func);
+      if (Ret != RelaxRet::NotModified)
+        Modified = true;
+      // changed. Also if we don't inserted stubs we don't have to run
+      // fixBranches and it could lead to the shortening relaxed instructions.
+      if (static_cast<int>(Ret) & RelaxRet::StubsInserted && Func->isSimple())
     }
   } while (Modified);
   outs() << "BOLT-INFO: Inserted " << NumHotStubs
          << " stubs in the hot area and " << NumColdStubs
          << " stubs in the cold area. Shared " << NumSharedStubs
          << " times, iterated " << Iterations << " times.\n";
 }
 } // namespace bolt
 } // namespace llvm

bolt/lib/Passes/ReorderFunctions.cpp

 //===- bolt/Passes/ReorderFunctions.cpp - Function reordering pass --------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements ReorderFunctions class.
 //
 //===----------------------------------------------------------------------===//
 
 #include "bolt/Passes/ReorderFunctions.h"
+#include "bolt/Passes/Golang.h"
 #include "bolt/Passes/HFSort.h"
 #include "bolt/Utils/Utils.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/Support/CommandLine.h"
 #include <fstream>
 
 #define DEBUG_TYPE "hfsort"
 
 using namespace llvm;
+using namespace bolt;
 
 namespace opts {
 
 extern cl::OptionCategory BoltOptCategory;
 extern cl::opt<unsigned> Verbosity;
 extern cl::opt<uint32_t> RandomSeed;
 
 extern size_t padFunction(const bolt::BinaryFunction &Function);

 using NodeId = CallGraph::NodeId;
 using Arc = CallGraph::Arc;
 using Node = CallGraph::Node;
 
 void ReorderFunctions::reorder(std::vector<Cluster> &&Clusters,
                                std::map<uint64_t, BinaryFunction> &BFs) {
   std::vector<uint64_t> FuncAddr(Cg.numNodes()); // Just for computing stats
   uint64_t TotalSize = 0;
-  uint32_t Index = 0;
+  uint32_t Index = FIRST_BF_INDEX;
 
   // Set order of hot functions based on clusters.
   for (const Cluster &Cluster : Clusters) {
     for (const NodeId FuncId : Cluster.targets()) {
       Cg.nodeIdToFunc(FuncId)->setIndex(Index++);
       FuncAddr[FuncId] = TotalSize;
       TotalSize += Cg.size(FuncId);
     }
void ReorderFunctions::runOnFunctions(BinaryContext &BC) {
   std::vector<Cluster> Clusters;
 
   switch (opts::ReorderFunctions) {
   case RT_NONE:
     break;
   case RT_EXEC_COUNT:
     {
       std::vector<BinaryFunction *> SortedFunctions(BFs.size());
-      uint32_t Index = 0;
+      uint32_t Index = FIRST_BF_INDEX;
       llvm::transform(llvm::make_second_range(BFs), SortedFunctions.begin(),
                       [](BinaryFunction &BF) { return &BF; });
       llvm::stable_sort(SortedFunctions, [&](const BinaryFunction *A,
                                              const BinaryFunction *B) {
         if (A->isIgnored())
           return false;
         const size_t PadA = opts::padFunction(*A);
         const size_t PadB = opts::padFunction(*B);
case RT_USER:
         errs() << "BOLT-WARNING: Reorder functions: can't find functions for "
                << InvalidEntries << " entries in -function-order list\n";
     }
     break;
   }
 
   reorder(std::move(Clusters), BFs);
 
+  if (opts::GolangPass != opts::GV_NONE && opts::ReorderFunctions != RT_USER) {
+    // | 0: runtime.text                           |
+    // | sorted functions                          |
+    // | -3: unsorted functions                    |
+    // | -2: runtime.etext                         |
+    // | -1: injected functions(including startup) |
+
+    for (auto &I : BFs) {
+      auto BF = &I.second;
+      if (BF->hasRestoredNameRegex(GolangPass::getFirstBFName())) {
+        BF->setIndex(GO_FIRST_BF_INDEX);
+      } else if (BF->hasRestoredNameRegex(GolangPass::getLastBFName())) {
+        BF->setIndex(GO_LAST_BF_INDEX);
+      } else if (!BF->isGolang()) {
+        BF->setIndex(INVALID_BF_INDEX);
+      } else if (!BF->hasValidIndex()) {
+        BF->setIndex(GO_UNUSED_BF_INDEX);
+      }
+    }
+  }
+
   std::unique_ptr<std::ofstream> FuncsFile;
   if (!opts::GenerateFunctionOrderFile.empty()) {
     FuncsFile = std::make_unique<std::ofstream>(opts::GenerateFunctionOrderFile,
                                                 std::ios::out);
     if (!FuncsFile) {
       errs() << "BOLT-ERROR: ordered functions file "
              << opts::GenerateFunctionOrderFile << " cannot be opened\n";
       exit(1);

bolt/lib/Passes/SplitFunctions.cpp

 #include <random>
 #include <vector>
 
 #define DEBUG_TYPE "bolt-opts"
 
 using namespace llvm;
 using namespace bolt;
 
-namespace {
 class DeprecatedSplitFunctionOptionParser : public cl::parser<bool> {
 public:
   explicit DeprecatedSplitFunctionOptionParser(cl::Option &O)
       : cl::parser<bool>(O) {}
 
   bool parse(cl::Option &O, StringRef ArgName, StringRef Arg, bool &Value) {
     if (Arg == "2" || Arg == "3") {
       Value = true;
       errs() << formatv("BOLT-WARNING: specifying non-boolean value \"{0}\" "
                         "for option -{1} is deprecated\n",
                         Arg, ArgName);
       return false;
     }
     return cl::parser<bool>::parse(O, ArgName, Arg, Value);
   }
 };
-} // namespace
 
 namespace opts {
 
 extern cl::OptionCategory BoltOptCategory;
 
 extern cl::opt<bool> SplitEH;
 extern cl::opt<unsigned> ExecutionCountThreshold;
 extern cl::opt<uint32_t> RandomSeed;
 
 static cl::opt<bool> AggressiveSplitting(
     "split-all-cold", cl::desc("outline as many cold basic blocks as possible"),
     cl::cat(BoltOptCategory));
 
 static cl::opt<unsigned> SplitAlignThreshold(
     "split-align-threshold",
     cl::desc("when deciding to split a function, apply this alignment "
              "while doing the size comparison (see -split-threshold). "
              "Default value: 2."),
     cl::init(2),
 
     cl::Hidden, cl::cat(BoltOptCategory));
 
-static cl::opt<bool, false, DeprecatedSplitFunctionOptionParser>
+cl::opt<bool, false, DeprecatedSplitFunctionOptionParser>
     SplitFunctions("split-functions",
                    cl::desc("split functions into fragments"),
                    cl::cat(BoltOptCategory));
 
 static cl::opt<unsigned> SplitThreshold(
     "split-threshold",
     cl::desc("split function only if its main size is reduced by more than "
              "given amount of bytes. Default value: 0, i.e. split iff the "

bolt/lib/Rewrite/BinaryPassManager.cpp

 //===- bolt/Rewrite/BinaryPassManager.cpp - Binary-level pass manager -----===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "bolt/Rewrite/BinaryPassManager.h"
 #include "bolt/Passes/ADRRelaxationPass.h"
 #include "bolt/Passes/Aligner.h"
 #include "bolt/Passes/AllocCombiner.h"
 #include "bolt/Passes/AsmDump.h"
 #include "bolt/Passes/CMOVConversion.h"
 #include "bolt/Passes/FixRelaxationPass.h"
 #include "bolt/Passes/FrameOptimizer.h"
+#include "bolt/Passes/Golang.h"
 #include "bolt/Passes/Hugify.h"
 #include "bolt/Passes/IdenticalCodeFolding.h"
 #include "bolt/Passes/IndirectCallPromotion.h"
 #include "bolt/Passes/Inliner.h"
 #include "bolt/Passes/Instrumentation.h"
 #include "bolt/Passes/JTFootprintReduction.h"
 #include "bolt/Passes/LongJmp.h"
 #include "bolt/Passes/LoopInversionPass.h"
SpecializeMemcpy1("memcpy1-spec",
   cl::desc("list of functions with call sites for which to specialize memcpy() "
            "for size 1"),
   cl::value_desc("func1,func2:cs1:cs2,func3:cs1,..."),
   cl::ZeroOrMore, cl::cat(BoltOptCategory));
 
 static cl::opt<bool> Stoke("stoke", cl::desc("turn on the stoke analysis"),
                            cl::cat(BoltOptCategory));
 
-static cl::opt<bool> StringOps(
+cl::opt<bool> StringOps(
     "inline-memcpy",
     cl::desc("inline memcpy using 'rep movsb' instruction (X86-only)"),
-    cl::cat(BoltOptCategory));
+    cl::init(false), cl::ZeroOrMore, cl::cat(BoltOptCategory));
 
 static cl::opt<bool> StripRepRet(
     "strip-rep-ret",
     cl::desc("strip 'repz' prefix from 'repz retq' sequence (on by default)"),
     cl::init(true), cl::cat(BoltOptCategory));
 
 static cl::opt<bool> VerifyCFG("verify-cfg",
                                cl::desc("verify the CFG after every pass"),
                                cl::Hidden, cl::cat(BoltOptCategory));
 
 static cl::opt<bool> ThreeWayBranchFlag("three-way-branch",
                                         cl::desc("reorder three way branches"),
                                         cl::ReallyHidden,
                                         cl::cat(BoltOptCategory));
 
 static cl::opt<bool> CMOVConversionFlag("cmov-conversion",
                                         cl::desc("fold jcc+mov into cmov"),
                                         cl::ReallyHidden,
                                         cl::cat(BoltOptCategory));
 
+cl::opt<bool> InstructionsLowering("lower-instructions",
+                                   cl::desc("Instructions lowering pass"),
+                                   cl::init(true), cl::ZeroOrMore, cl::Hidden,
+                                   cl::cat(BoltCategory));
+
 } // namespace opts
 
 namespace llvm {
 namespace bolt {
 
 using namespace opts;
 
 const char BinaryFunctionPassManager::TimerGroupName[] = "passman";
void BinaryFunctionPassManager::runAllPasses(BinaryContext &BC) {
   // order they're registered.
 
   // Run this pass first to use stats for the original functions.
   Manager.registerPass(std::make_unique<PrintProgramStats>(NeverPrint));
 
   if (opts::PrintProfileStats)
     Manager.registerPass(std::make_unique<PrintProfileStats>(NeverPrint));
 
+  if (opts::GolangPass != opts::GV_NONE)
+    Manager.registerPass(std::make_unique<GolangPrePass>(BC));
+
   Manager.registerPass(std::make_unique<ValidateInternalCalls>(NeverPrint));
 
   Manager.registerPass(std::make_unique<ValidateMemRefs>(NeverPrint));
 
   if (opts::Instrument)
     Manager.registerPass(std::make_unique<Instrumentation>(NeverPrint));
   else if (opts::Hugify)
     Manager.registerPass(std::make_unique<HugePage>(NeverPrint));
void BinaryFunctionPassManager::runAllPasses(BinaryContext &BC) {
   Manager.registerPass(std::make_unique<Peepholes>(PrintPeepholes));
 
   Manager.registerPass(std::make_unique<AlignerPass>());
 
   // Perform reordering on data contained in one or more sections using
   // memory profiling data.
   Manager.registerPass(std::make_unique<ReorderData>());
 
-  if (BC.isAArch64()) {
+  if (BC.isAArch64())
     Manager.registerPass(std::make_unique<ADRRelaxationPass>());
 
+  if (opts::GolangPass != opts::GV_NONE)
+    Manager.registerPass(std::make_unique<GolangPostPass>(BC));
+
+  if (BC.isAArch64() || opts::GolangPass != opts::GV_NONE) {
     // Tighten branches according to offset differences between branch and
     // targets. No extra instructions after this pass, otherwise we may have
     // relocations out of range and crash during linking.
     Manager.registerPass(std::make_unique<LongJmpPass>(PrintLongJmp));
   }
 
   // This pass should always run last.*
   Manager.registerPass(std::make_unique<FinalizeFunctions>(PrintFinalized));
void BinaryFunctionPassManager::runAllPasses(BinaryContext &BC) {
   // Patch original function entries
   if (BC.HasRelocations)
     Manager.registerPass(std::make_unique<PatchEntries>());
 
   // This pass turns tail calls into jumps which makes them invisible to
   // function reordering. It's unsafe to use any CFG or instruction analysis
   // after this point.
   Manager.registerPass(
-      std::make_unique<InstructionLowering>(PrintAfterLowering));
+      std::make_unique<InstructionLowering>(PrintAfterLowering),
+      opts::InstructionsLowering);
 
   // In non-relocation mode, mark functions that do not fit into their original
   // space as non-simple if we have to (e.g. for correct debug info update).
   // NOTE: this pass depends on finalized code.
   if (!BC.HasRelocations)
     Manager.registerPass(std::make_unique<CheckLargeFunctions>(NeverPrint));
 
+  if (opts::GolangPass != opts::GV_NONE)
+    Manager.registerPass(std::make_unique<GolangPass>(BC));
+
   Manager.registerPass(std::make_unique<LowerAnnotations>(NeverPrint));
 
   Manager.runPasses();
 }
 
 } // namespace bolt
 } // namespace llvm

bolt/lib/Rewrite/RewriteInstance.cpp

 #include "bolt/Core/BinaryEmitter.h"
 #include "bolt/Core/BinaryFunction.h"
 #include "bolt/Core/DebugData.h"
 #include "bolt/Core/Exceptions.h"
 #include "bolt/Core/FunctionLayout.h"
 #include "bolt/Core/MCPlusBuilder.h"
 #include "bolt/Core/ParallelUtilities.h"
 #include "bolt/Core/Relocation.h"
+#include "bolt/Passes/BinaryPasses.h"
 #include "bolt/Passes/CacheMetrics.h"
 #include "bolt/Passes/ReorderFunctions.h"
+#include "bolt/Passes/SplitFunctions.h"
 #include "bolt/Profile/BoltAddressTranslation.h"
 #include "bolt/Profile/DataAggregator.h"
 #include "bolt/Profile/DataReader.h"
 #include "bolt/Profile/YAMLProfileReader.h"
 #include "bolt/Profile/YAMLProfileWriter.h"
 #include "bolt/Rewrite/BinaryPassManager.h"
 #include "bolt/Rewrite/DWARFRewriter.h"
 #include "bolt/Rewrite/ExecutableFileMemoryManager.h"

 extern cl::opt<MacroFusionType> AlignMacroOpFusion;
 extern cl::list<std::string> HotTextMoveSections;
 extern cl::opt<bool> Hugify;
 extern cl::opt<bool> Instrument;
 extern cl::opt<JumpTableSupportLevel> JumpTables;
 extern cl::list<std::string> ReorderData;
 extern cl::opt<bolt::ReorderFunctions::ReorderType> ReorderFunctions;
 extern cl::opt<bool> TimeBuild;
+extern cl::opt<bool> PreserveBlocksAlignment;
+extern cl::opt<bool> AlignBlocks;
+extern cl::opt<bool> NoInline;
+extern cl::opt<bool> StringOps;
+extern cl::opt<FrameOptimizationType> FrameOptimization;
+extern cl::opt<bool> SplitFunctions;
+extern llvm::cl::opt<bool> InsertRetpolines;
+extern cl::opt<bool> InstructionsLowering;
+extern cl::list<Peepholes::PeepholeOpts> Peepholes;
 
 cl::opt<bool> AllowStripped("allow-stripped",
                             cl::desc("allow processing of stripped binaries"),
                             cl::Hidden, cl::cat(BoltCategory));
 
 static cl::opt<bool> ForceToDataRelocations(
     "force-data-relocations",
     cl::desc("force relocations to data sections to always be processed"),
void RewriteInstance::adjustCommandLineOptions() {
   }
 
   if (opts::HotText && opts::HotTextMoveSections.getNumOccurrences() == 0) {
     opts::HotTextMoveSections.addValue(".stub");
     opts::HotTextMoveSections.addValue(".mover");
     opts::HotTextMoveSections.addValue(".never_hugify");
   }
 
+  if ((BC->isAArch64() || opts::GolangPass != opts::GV_NONE) &&
+      (opts::AlignBlocks || opts::PreserveBlocksAlignment)) {
+    errs() << "BOLT-WARNING: Disabling block alignment\n";
+    opts::AlignBlocks = false;
+    opts::PreserveBlocksAlignment = false;
+  }
+
+  if (opts::GolangPass != opts::GV_NONE) {
+    // Golang does not support inlining
+    opts::NoInline = true;
+    opts::StringOps = false;
+
+    // Instructions should not be lowered for golang
+    opts::InstructionsLowering = false;
+
+    // Enable double jump elimination. Currently golang uses them heavely and
+    // this option is needed for preserving pcsp tables values correctly.
+    opts::Peepholes.push_back(Peepholes::PEEP_DOUBLE_JUMPS);
+
+    if (opts::FrameOptimization != FOP_NONE) {
+      errs() << "BOLT-WARNING: Golang does not support frame optimizations\n";
+      opts::FrameOptimization = FOP_NONE;
+    }
+
+    if (opts::SplitFunctions) {
+      errs() << "BOLT-WARNING: Golang does not support function splitting\n";
+      opts::SplitFunctions = false;
+    }
+
+    if (opts::UseOldText) {
+      errs() << "BOLT-WARNING: Cannot combine -use-old-text and -golang\n";
+      opts::UseOldText = false;
+    }
+
+    if (opts::Lite) {
+      errs() << "BOLT-WARNING: Lite mode is not compatible with -golang. "
+                "Disabling.\n";
+      opts::Lite = false;
+    }
+
+    if (opts::HotFunctionsAtEnd) {
+      errs() << "BOLT-WARNING: Golang does not support hot functions at end. "
+                "Disabling.\n";
+      opts::HotFunctionsAtEnd = false;
+    }
+
+    if (opts::InsertRetpolines) {
+      errs() << "BOLT-WARNING: Retpoline pass is not compatible with -golang. "
+                "Disabling.\n";
+      opts::InsertRetpolines = false;
+    }
+  }
+
   if (opts::UseOldText && !BC->OldTextSectionAddress) {
     errs() << "BOLT-WARNING: cannot use old .text as the section was not found"
               "\n";
     opts::UseOldText = false;
   }
   if (opts::UseOldText && !BC->HasRelocations) {
     errs() << "BOLT-WARNING: cannot use old .text in non-relocation mode\n";
     opts::UseOldText = false;
   }
 
   if (!opts::AlignText.getNumOccurrences())
     opts::AlignText = BC->PageAlign;
 
   if (opts::AlignText < opts::AlignFunctions)
     opts::AlignText = (unsigned)opts::AlignFunctions;
 
   if (BC->isX86() && opts::Lite.getNumOccurrences() == 0 && !opts::StrictMode &&
-      !opts::UseOldText)
+      !opts::UseOldText && opts::GolangPass == opts::GV_NONE)
     opts::Lite = true;
 
   if (opts::Lite && opts::UseOldText) {
     errs() << "BOLT-WARNING: cannot combine -lite with -use-old-text. "
               "Disabling -use-old-text.\n";
     opts::UseOldText = false;
   }
 
void RewriteInstance::processProfileData() {
 
   if (opts::AggregateOnly)
     exit(0);
 }
 
 void RewriteInstance::disassembleFunctions() {
   NamedRegionTimer T("disassembleFunctions", "disassemble functions",
                      TimerGroupName, TimerGroupDesc, opts::TimeRewrite);
+  // Create annotation indices to allow lock-free execution
+  BC->MIB->getOrCreateAnnotationIndex("Size");
+  BC->MIB->getOrCreateAnnotationIndex("Locked");
+
   for (auto &BFI : BC->getBinaryFunctions()) {
     BinaryFunction &Function = BFI.second;
 
     ErrorOr<ArrayRef<uint8_t>> FunctionData = Function.getData();
     if (!FunctionData) {
       errs() << "BOLT-ERROR: corresponding section is non-executable or "
              << "empty for function " << Function << '\n';
       exit(1);
if (!ELF64LEFile) {
     errs() << "BOLT-ERROR: only 64-bit LE ELF binaries are supported\n";
     exit(1);
   }
   const ELFFile<ELF64LE> &Obj = ELF64LEFile->getELFFile();
   raw_fd_ostream &OS = Out->os();
 
   // Write/re-write program headers.
   Phnum = Obj.getHeader().e_phnum;
+
   if (PHDRTableOffset) {
     // Writing new pheader table and adding one new entry for R+X segment.
     Phnum += 1;
     if (NewWritableSegmentSize) {
       // Adding one more entry for R+W segment.
       Phnum += 1;
     }
   } else {
if (PHDRTableAddress && Phdr.p_type == ELF::PT_PHDR) {
                sizeof(NewTextPhdr));
       if (NewWritableSegmentSize) {
         ELF64LEPhdrTy NewWritablePhdr = createNewWritableSectionsPhdr();
         OS.write(reinterpret_cast<const char *>(&NewWritablePhdr),
                  sizeof(NewWritablePhdr));
       }
       AddedSegment = true;
     }
+
     OS.write(reinterpret_cast<const char *>(&NewPhdr), sizeof(NewPhdr));
   }
 
   if (!opts::UseGnuStack && !AddedSegment) {
     // Append the new header to the end of the table.
     ELF64LE::Phdr NewTextPhdr = createNewTextPhdr();
     OS.write(reinterpret_cast<const char *>(&NewTextPhdr), sizeof(NewTextPhdr));
     if (NewWritableSegmentSize) {
if (Function) {
               "Cannot locate fragment containg secondary entrypoint");
           FF = Function->getLayout().fragment_begin();
         }
 
         NewSymbol.st_shndx =
             Function->getCodeSection(FF->getFragmentNum())->getIndex();
       } else {
         // Check if the symbol belongs to moved data object and update it.
-        BinaryData *BD = opts::ReorderData.empty()
-                             ? nullptr
-                             : BC->getBinaryDataAtAddress(Symbol.st_value);
+        BinaryData *BD =
+            opts::ForceToDataRelocations || !opts::ReorderData.empty()
+                ? BC->getBinaryDataAtAddress(Symbol.st_value)
+                : nullptr;
         if (BD && BD->isMoved() && !BD->isJumpTable()) {
           assert((!BD->getSize() || !Symbol.st_size ||
                   Symbol.st_size == BD->getSize()) &&
                  "sizes must match");
 
           BinarySection &OutputSection = BD->getOutputSection();
           assert(OutputSection.getIndex());
           LLVM_DEBUG(dbgs()
                      << "BOLT-DEBUG: moving " << BD->getName() << " from "
                      << *BC->getSectionNameForAddress(Symbol.st_value) << " ("
                      << Symbol.st_shndx << ") to " << OutputSection.getName()
                      << " (" << OutputSection.getIndex() << ")\n");
           NewSymbol.st_shndx = OutputSection.getIndex();
           NewSymbol.st_value = BD->getOutputAddress();
+          NewSymbol.st_size = BD->getOutputSize();
         } else {
           // Otherwise just update the section for the symbol.
           if (Symbol.st_shndx < ELF::SHN_LORESERVE)
             NewSymbol.st_shndx = getNewSectionIndex(Symbol.st_shndx);
         }
 
         // Detect local syms in the text section that we didn't update
         // and that were preserved by the linker to support relocations against
if (opts::HotText && !NumHotTextSymsUpdated) {
     addSymbol("__hot_end");
   }
 
   if (opts::HotData && !NumHotDataSymsUpdated) {
     addSymbol("__hot_data_start");
     addSymbol("__hot_data_end");
   }
 
+  auto addSectionSymbol = [&](uint64_t Address, unsigned index) {
+    ELFSymTy Symbol;
+    Symbol.st_value = Address;
+    Symbol.st_shndx = index;
+    Symbol.st_name = AddToStrTab("");
+    Symbol.st_size = 0;
+    Symbol.st_other = 0;
+    Symbol.setBindingAndType(ELF::STB_LOCAL, ELF::STT_SECTION);
+
+    Symbols.insert(Symbols.begin() + index, Symbol);
+  };
+
+  for (BinarySection &Section : BC->allocatableSections()) {
+    if (Section.hasSectionRef())
+      continue;
+
+    addSectionSymbol(Section.getOutputAddress(), Section.getIndex());
+  }
+
   // Put local symbols at the beginning.
   llvm::stable_sort(Symbols, [](const ELFSymTy &A, const ELFSymTy &B) {
     if (A.getBinding() == ELF::STB_LOCAL && B.getBinding() != ELF::STB_LOCAL)
       return true;
     return false;
   });
 
   for (const ELFSymTy &Symbol : Symbols)
for (BinarySection &Section : BC->allocatableSections()) {
         Elf_Rela NewRelA;
         MCSymbol *Symbol = Rel.Symbol;
         uint32_t SymbolIdx = 0;
         uint64_t Addend = Rel.Addend;
         uint64_t RelOffset =
             getNewFunctionOrDataAddress(SectionInputAddress + Rel.Offset);
 
         RelOffset = RelOffset == 0 ? SectionAddress + Rel.Offset : RelOffset;
-        if (Rel.Symbol) {
+        if (Symbol) {
+          if (!IsRelative && !Rel.isIRelative()) {
             SymbolIdx = getOutputDynamicSymbolIndex(Symbol);
           } else {
+            // The R_*_(I)RELATIVE relocation inserted by BOLT
+            ErrorOr<uint64_t> Address = BC->getSymbolValue(*Symbol);
+            if (Address)
+              Addend += getNewFunctionOrDataAddress(*Address);
+          }
+        } else {
           // Usually this case is used for R_*_(I)RELATIVE relocations
           const uint64_t Address = getNewFunctionOrDataAddress(Addend);
           if (Address)
             Addend = Address;
         }
 
         NewRelA.setSymbolAndType(SymbolIdx, Rel.Type, EF.isMips64EL());
         NewRelA.r_offset = RelOffset;

bolt/lib/RuntimeLibs/InstrumentationRuntimeLibrary.cpp

if (opts::JumpTables != JTS_MOVE) {
     outs() << "BOLT-INFO: forcing -jump-tables=move for instrumentation\n";
   }
   if (!BC.StartFunctionAddress) {
     errs() << "BOLT-ERROR: instrumentation runtime libraries require a known "
               "entry point of "
               "the input binary\n";
     exit(1);
   }
-  if (!BC.FiniFunctionAddress && !BC.IsStaticExecutable) {
+  if (!BC.FiniFunctionAddress && !BC.IsStaticExecutable &&
+      opts::GolangPass == opts::GV_NONE) {
     errs() << "BOLT-ERROR: input binary lacks DT_FINI entry in the dynamic "
               "section but instrumentation currently relies on patching "
               "DT_FINI to write the profile\n";
     exit(1);
   }
 }
 
 void InstrumentationRuntimeLibrary::emitBinary(BinaryContext &BC,
void InstrumentationRuntimeLibrary::emitBinary(BinaryContext &BC,
   emitIntValue("__bolt_instr_num_ind_calls",
                Summary->IndCallDescriptions.size());
   emitIntValue("__bolt_instr_num_ind_targets",
                Summary->IndCallTargetDescriptions.size());
   emitIntValue("__bolt_instr_num_funcs", Summary->FunctionDescriptions.size());
   emitString("__bolt_instr_filename", opts::InstrumentationFilename);
   emitString("__bolt_instr_binpath", opts::InstrumentationBinpath);
   emitIntValue("__bolt_instr_use_pid", !!opts::InstrumentationFileAppendPID, 1);
+  emitValue(BC.Ctx->getOrCreateSymbol("__bolt_trampoline_instr_fini_call"),
+            nullptr);
 
   if (BC.isMachO()) {
     MCSection *TablesSection = BC.Ctx->getMachOSection(
         "__BOLT", "__tables", MachO::S_REGULAR, SectionKind::getData());
     TablesSection->setAlignment(llvm::Align(BC.RegularPageSize));
     Streamer.switchSection(TablesSection);
     emitString("__bolt_instr_tables", buildTables(BC));
   }

bolt/lib/Target/AArch64/AArch64MCPlusBuilder.cpp

public:
   bool convertTailCallToJmp(MCInst &Inst) override {
     removeAnnotation(Inst, MCPlus::MCAnnotation::kTailCall);
     clearOffset(Inst);
     if (getConditionalTailCall(Inst))
       unsetConditionalTailCall(Inst);
     return true;
   }
 
+  bool relaxInstruction(MCInst &Inst) const override {
+    (void)Inst;
+    return false;
+  }
+
   bool lowerTailCall(MCInst &Inst) override {
     removeAnnotation(Inst, MCPlus::MCAnnotation::kTailCall);
     if (getConditionalTailCall(Inst))
       unsetConditionalTailCall(Inst);
     return true;
   }
 
   bool isNoop(const MCInst &Inst) const override {

bolt/lib/Target/X86/X86MCPlusBuilder.cpp

default:
     return Opcode;
   case X86::JMP_2: return X86::JMP_1;
   case X86::JMP_4: return X86::JMP_1;
   case X86::JCC_2: return X86::JCC_1;
   case X86::JCC_4: return X86::JCC_1;
   }
 }
 
+unsigned getLongBranchOpcode(unsigned Opcode) {
+  switch (Opcode) {
+  default:
+    return Opcode;
+  case X86::JMP_1:
+    return X86::JMP_4;
+  case X86::JMP_2:
+    return X86::JMP_4;
+  case X86::JCC_1:
+    return X86::JCC_4;
+  case X86::JCC_2:
+    return X86::JCC_4;
+  }
+}
+
 unsigned getShortArithOpcode(unsigned Opcode) {
   return X86::getShortOpcodeArith(Opcode);
 }
 
 bool isMOVSX64rm32(const MCInst &Inst) {
   return Inst.getOpcode() == X86::MOVSX64rm32;
 }
 
if (I.IsLoad) {
         int64_t SrcImm = SrcOpnd.getImm();
         Inst.clear();
         Inst.setOpcode(NewOpcode);
         Inst.addOperand(MCOperand::createImm(SrcImm));
       }
     }
   }
 
+  int getStackAdjustment(const MCInst &Inst) const override {
+    int Ret;
+
+    if ((Ret = getPushSize(Inst))) {
+      return Ret;
+    } else if ((Ret = getPopSize(Inst))) {
+      return -Ret;
+    }
+
+    const int ValOp = 2;
+    int Sign = 1;
+
+    switch (Inst.getOpcode()) {
+    default:
+      return 0;
+    case X86::SUB64ri32:
+    case X86::SUB64ri8:
+      break;
+    case X86::ADD64ri32:
+    case X86::ADD64ri8:
+      Sign = -1;
+      break;
+    }
+
+    const MCInstrDesc &MCII = Info->get(Inst.getOpcode());
+    for (int I = 0, E = MCII.getNumDefs(); I != E; ++I) {
+      const MCOperand &Operand = Inst.getOperand(I);
+      if (Operand.isReg() && Operand.getReg() == X86::RSP) {
+        assert(Inst.getOperand(ValOp).isImm() && "unexpected operand");
+        return (int)Inst.getOperand(ValOp).getImm() * Sign;
+      }
+    }
+
+    return 0;
+  }
+
   bool isStackAdjustment(const MCInst &Inst) const override {
     switch (Inst.getOpcode()) {
     default:
       return false;
     case X86::SUB64ri32:
     case X86::SUB64ri8:
     case X86::ADD64ri32:
     case X86::ADD64ri8:
if (isBranch(Inst)) {
       NewOpcode = getShortBranchOpcode(OldOpcode);
     } else if (OldOpcode == X86::MOV64ri) {
       if (Inst.getOperand(MCPlus::getNumPrimeOperands(Inst) - 1).isImm()) {
         const int64_t Imm =
             Inst.getOperand(MCPlus::getNumPrimeOperands(Inst) - 1).getImm();
         if (int64_t(Imm) == int64_t(int32_t(Imm)))
           NewOpcode = X86::MOV64ri32;
       }
+    } else if (isNoop(Inst) && hasAnnotation(Inst, "NOP")) {
+      NewOpcode = X86::NOOP;
     } else {
       // If it's arithmetic instruction check if signed operand fits in 1 byte.
       const unsigned ShortOpcode = getShortArithOpcode(OldOpcode);
       if (ShortOpcode != OldOpcode &&
           Inst.getOperand(MCPlus::getNumPrimeOperands(Inst) - 1).isImm()) {
         int64_t Imm =
             Inst.getOperand(MCPlus::getNumPrimeOperands(Inst) - 1).getImm();
         if (int64_t(Imm) == int64_t(int8_t(Imm)))
convertMoveToConditionalMove(MCInst &Inst, unsigned CC, bool AllowStackMemOp,
     // Insert CC at the end of prime operands, before annotations
     Inst.insert(Inst.begin() + MCPlus::getNumPrimeOperands(Inst),
                 MCOperand::createImm(CC));
     // CMOV is a 3-operand MCInst, so duplicate the destination as src1
     Inst.insert(Inst.begin(), Inst.getOperand(0));
     return true;
   }
 
+  bool relaxInstruction(MCInst &Inst) const override {
+    unsigned OldOpcode = Inst.getOpcode();
+    unsigned NewOpcode = OldOpcode;
+
+    if (isBranch(Inst) || isTailCall(Inst)) {
+      NewOpcode = getLongBranchOpcode(OldOpcode);
+    }
+
+    if (NewOpcode == OldOpcode)
+      return false;
+
+    Inst.setOpcode(NewOpcode);
+    return true;
+  }
+
   bool lowerTailCall(MCInst &Inst) override {
     if (Inst.getOpcode() == X86::JMP_4 && isTailCall(Inst)) {
       Inst.setOpcode(X86::JMP_1);
       removeAnnotation(Inst, MCPlus::MCAnnotation::kTailCall);
       return true;
     }
     return false;
   }
bool analyzeBranch(InstructionIterator Begin, InstructionIterator End,
                      MCInst *&UncondBranch) const override {
     auto I = End;
 
     // Bottom-up analysis
     while (I != Begin) {
       --I;
 
       // Ignore nops and CFIs
-      if (isPseudo(*I))
+      if (isPseudo(*I) || isNoop(*I))
         continue;
 
       // Stop when we find the first non-terminator
       if (!isTerminator(*I))
         break;
 
       if (!isBranch(*I))
         break;
bool createIJmp32Frag(SmallVectorImpl<MCInst> &Insts,
     Load.addOperand(MCOperand::createReg(X86::NoRegister));
 
     Insts.push_back(Load);
     Insts.push_back(IJmp);
     return true;
   }
 
   bool createNoop(MCInst &Inst) const override {
+    Inst.clear();
     Inst.setOpcode(X86::NOOP);
     return true;
   }
 
   bool createReturn(MCInst &Inst) const override {
     Inst.setOpcode(X86::RET64);
     return true;
   }

   bool createUncondBranch(MCInst &Inst, const MCSymbol *TBB,
                           MCContext *Ctx) const override {
     Inst.setOpcode(X86::JMP_1);
     Inst.addOperand(MCOperand::createExpr(
         MCSymbolRefExpr::create(TBB, MCSymbolRefExpr::VK_None, *Ctx)));
     return true;
   }
 
+  int getUncondBranchEncodingSize() const override { return 8; }
+
   bool createCall(MCInst &Inst, const MCSymbol *Target,
                   MCContext *Ctx) override {
     Inst.setOpcode(X86::CALL64pcrel32);
     Inst.addOperand(MCOperand::createExpr(
         MCSymbolRefExpr::create(Target, MCSymbolRefExpr::VK_None, *Ctx)));
     return true;
   }
 
+  int getShortJmpEncodingSize() const override { return 8; }
+
+  int getPCRelEncodingSize(const MCInst &Inst) const override {
+    switch (Inst.getOpcode()) {
+    default:
+      llvm_unreachable("Failed to get pcrel encoding size");
+      return 0;
+    case X86::JMP_1:
+    case X86::JCC_1:
+      return 8;
+    case X86::JMP_2:
+    case X86::JCC_2:
+    case X86::JMP16m:
+    case X86::JMP16r:
+      return 16;
+    case X86::JMP_4:
+    case X86::JCC_4:
+    case X86::JMP32m:
+    case X86::JMP32r:
+    case X86::CALL64pcrel32:
+      return 32;
+    case X86::JMP64m:
+    case X86::JMP64r:
+    case X86::CALL64m:
+    case X86::CALL64r:
+      return 64;
+    }
+  }
+
   bool createTailCall(MCInst &Inst, const MCSymbol *Target,
                       MCContext *Ctx) override {
     return createDirectCall(Inst, Target, Ctx, /*IsTailCall*/ true);
   }
 
   void createLongTailCall(InstructionListType &Seq, const MCSymbol *Target,
                           MCContext *Ctx) override {
     Seq.clear();
void createIndirectBranch(MCInst &Inst, MCPhysReg MemBaseReg,
     Inst.setOpcode(X86::JMP64m);
     Inst.addOperand(MCOperand::createReg(MemBaseReg));      // BaseReg
     Inst.addOperand(MCOperand::createImm(1));               // ScaleAmt
     Inst.addOperand(MCOperand::createReg(X86::NoRegister)); // IndexReg
     Inst.addOperand(MCOperand::createImm(Disp));            // Displacement
     Inst.addOperand(MCOperand::createReg(X86::NoRegister)); // AddrSegmentReg
   }
 
+  bool createIndirectCall(MCInst &Inst, const MCSymbol *TargetLocation,
+                          MCContext *Ctx, bool IsTailCall) {
+    Inst.setOpcode(IsTailCall ? X86::JMP32m : X86::CALL64m);
+    Inst.addOperand(MCOperand::createReg(X86::RIP));        // BaseReg
+    Inst.addOperand(MCOperand::createImm(1));               // ScaleAmt
+    Inst.addOperand(MCOperand::createReg(X86::NoRegister)); // IndexReg
+    Inst.addOperand(MCOperand::createExpr(                  // Displacement
+        MCSymbolRefExpr::create(TargetLocation, MCSymbolRefExpr::VK_None,
+                                *Ctx)));
+    Inst.addOperand(MCOperand::createReg(X86::NoRegister)); // AddrSegmentReg
+    if (IsTailCall)
+      setTailCall(Inst);
+    return true;
+  }
+
+  InstructionListType createInstrumentFiniCall(MCSymbol *HandlerFuncAddr,
+                                               MCContext *Ctx,
+                                               bool IsTailCall) override {
+    // NOTE: We don't have to check HandlerFuncAddr content for 0 before call
+    // at runtime since Golang does't call any constructors
+    std::vector<MCInst> Insts(1);
+    createIndirectCall(Insts[0], HandlerFuncAddr, Ctx, IsTailCall);
+    addAnnotation(Insts[0], "IsInstrumentation", true);
+    return Insts;
+  }
+
   InstructionListType createInstrumentedIndirectCall(MCInst &&CallInst,
                                                      MCSymbol *HandlerFuncAddr,
                                                      int CallSiteID,
                                                      MCContext *Ctx) override {
     // Check if the target address expression used in the original indirect call
     // uses the stack pointer, which we are going to clobber.
     static BitVector SPAliases(getAliases(X86::RSP));
     bool UsesSP = false;

bolt/lib/Utils/CommandLineOpts.cpp

cl::opt<bool> UpdateDebugSections(
     cl::desc("update DWARF debug sections of the executable"),
     cl::cat(BoltCategory));
 
 cl::opt<unsigned>
     Verbosity("v", cl::desc("set verbosity level for diagnostic output"),
               cl::init(0), cl::ZeroOrMore, cl::cat(BoltCategory),
               cl::sub(cl::SubCommand::getAll()));
 
+cl::opt<opts::GolangVersion> GolangPass(
+    "golang",
+    cl::desc("Use for input binary generated with golang gc compiler"),
+    cl::init(opts::GV_NONE),
+    cl::values(
+        clEnumValN(opts::GV_NONE, "0", "do not use golang optimizations"),
+        clEnumValN(opts::GV_AUTO, "1", "auto detect golang version"),
+        clEnumValN(opts::GV_1_14_9, "1.14.9", "set gc version to 1.14.9"),
+        clEnumValN(opts::GV_1_14_12, "1.14.12", "set gc version to 1.14.12"),
+        clEnumValN(opts::GV_1_16_5, "1.16.5", "set gc version to 1.16.5"),
+        clEnumValN(opts::GV_1_17_2, "1.17.2", "set gc version to 1.17.2"),
+        clEnumValN(opts::GV_1_17_5, "1.17.5", "set gc version to 1.17.5"),
+        clEnumValN(opts::GV_1_17_8, "1.17.8", "set gc version to 1.17.8")),
+    cl::Optional, cl::cat(BoltOptCategory));
+
 bool processAllFunctions() {
   if (opts::AggregateOnly)
     return false;
 
   if (UseOldText || StrictMode)
     return true;
 
   return false;
 }
 
 } // namespace opts

bolt/runtime/instr.cpp

 // will identify indirect calls and modify them to load the address in these
 // trampolines and call this address instead. BOLT can't use direct calls to
 // our handlers because our addresses here are not known at analysis time. We
 // only support resolving dependencies from this file to the output of BOLT,
 // *not* the other way around.
 // TODO: We need better linking support to make that happen.
 extern void (*__bolt_ind_call_counter_func_pointer)();
 extern void (*__bolt_ind_tailcall_counter_func_pointer)();
+// Function pointer to __bolt_instr_fini
+extern void (*__bolt_trampoline_instr_fini_call)();
 // Function pointers to init/fini trampoline routines in the binary, so we can
 // resume regular execution of these functions that we hooked
 extern void __bolt_start_trampoline();
 extern void __bolt_fini_trampoline();
 
 #endif
 }
 

 
 out:;
   DEBUG(report("My parent process is dead, bye!\n"));
   __exit(0);
 }
 
 extern "C" void __bolt_instr_indirect_call();
 extern "C" void __bolt_instr_indirect_tailcall();
+extern "C" void __bolt_instr_fini();
 
 /// Initialization code
 extern "C" void __attribute((force_align_arg_pointer)) __bolt_instr_setup() {
   const uint64_t CountersStart =
       reinterpret_cast<uint64_t>(&__bolt_instr_locations[0]);
   const uint64_t CountersEnd = alignTo(
       reinterpret_cast<uint64_t>(&__bolt_instr_locations[__bolt_num_counters]),
       0x1000);
   DEBUG(reportNumber("replace mmap start: ", CountersStart, 16));
   DEBUG(reportNumber("replace mmap stop: ", CountersEnd, 16));
   assert (CountersEnd > CountersStart, "no counters");
   // Maps our counters to be shared instead of private, so we keep counting for
   // forked processes
   __mmap(CountersStart, CountersEnd - CountersStart,
          0x3 /*PROT_READ|PROT_WRITE*/,
          0x31 /*MAP_ANONYMOUS | MAP_SHARED | MAP_FIXED*/, -1, 0);
 
   __bolt_ind_call_counter_func_pointer = __bolt_instr_indirect_call;
   __bolt_ind_tailcall_counter_func_pointer = __bolt_instr_indirect_tailcall;
+  __bolt_trampoline_instr_fini_call = __bolt_instr_fini;
+
   // Conservatively reserve 100MiB shared pages
   GlobalAlloc.setMaxSize(0x6400000);
   GlobalAlloc.setShared(true);
   GlobalWriteProfileMutex = new (GlobalAlloc, 0) Mutex();
   if (__bolt_instr_num_ind_calls > 0)
     GlobalIndCallCounters =
         new (GlobalAlloc, 0) IndirectCallHashTable[__bolt_instr_num_ind_calls];