[BranchAlign] Compiler support for suppressing branch align

[reames]

Summary

As discussed heavily in the original review (D70157), there's a need for the compiler to be able to selective suppress padding (either nop or prefix) to respect assumptions about the meaning of labels and instructions in generated code.

Rather than wait for syntax to be finalized - which appears to be a very slow process - this patch focuses on the compiler use case and *only* worries about the integrated assembler. To my knowledge, this covers all cases mentioned to date for clang/JIT support.

For testing purposes, I wired it up so that if the integrated assembler was using autopadding for branch alignment (e.g. enabled at command line) then the textual assembly output would contain a comment for each location where padding was enabled or disabled. This seemed like the least painful choice overall.

Note that the result of this patch effective disables the jcc errata mitigation for many constructs (statepoints, implicit null checks, xray, etc...) which is non ideal. It is at least *correct* and should allow us to enable the mitigation for the compiler. Once that's done, and a few other items are worked through, we probably want to come back to this an explore a bundling based approach instead so that we can pad instructions while keeping labels in the right place.

[reames]

Really helps if I upload the patch correctly...

[reames]

It looks like the actual padding support got lost in one of the rebases. I'll need to rewrite that part. Though, I think I'm going to take an alternate approach which prioritizes the integrated assembler and simple marks nopadding regions with comments (i.e. rather than directives). That gives me most of what I'm looking for testing wise, and is a lot less likely to be controversial.

[reames]

Completely rework patch to support integrated assembler only + comments for testing.

(Warning: patch description heavily modified)

[MaskRay]

llvm/lib/MC/MCAsmStreamer.cpp

80

Some targets (XCore) do not have a MCAsmBackend (Target::hasMCAsmBackend() is false).

Assembler->getBackend() cannot be called.

Check test/MC/Disassembler/XCore/xcore.txt

[reames]

llvm/lib/MC/MCAsmStreamer.cpp

80

Oy, will make the call conditionally.

[MaskRay]

Generally good, only some suggestions for the tests.

llvm/lib/Target/X86/X86MCInstLower.cpp

1150

We could also be more specific. As an example, disabling auto padding only when --x86-align-branch=jcc is included & jcc is emitted by the relevant LowerXXXXX. The value is low, so being unconditional should be fine.

1281

Whitespace on an otherwise empty line.

Some other places in this file has such whitespace.

llvm/test/CodeGen/X86/align-branch-boundary-noautopadding.ll

1 ↗

(On Diff #236694)

This should be unnecessary.

2 ↗

(On Diff #236694)

Make llvm-objdump | FileCheck a separate step.

Add a separate assembly output test, checking #noautopadding is emitted.

18 ↗

(On Diff #236694)

Just check the intended offset sequence.

0:
1: callq
6: callq
b: callq
10: callq
15: callq
1a: callq
1f: callq

[reames]

llvm/lib/Target/X86/X86MCInstLower.cpp

1150

Sorry, was this a comment, or a request? I definitely do not want to make this any more complicated at this time. Only reason I bothered filtering the comments at all was to keep the test churn down.

llvm/test/CodeGen/X86/align-branch-boundary-noautopadding.ll

1 ↗

(On Diff #236694)

What specifically?

2 ↗

(On Diff #236694)

I'd really prefer not to introduce a temporary file here. It adds no value.

Happy to add the textual assembly check.

[MaskRay]

llvm/test/CodeGen/X86/align-branch-boundary-noautopadding.ll

1 ↗

(On Diff #236694)

Delete ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py

This is a hand-written test file.

2 ↗

(On Diff #236694)

I'd really prefer not to introduce a temporary file here. It adds no value.

OK. No strong opinion here.

[skan]

llvm/lib/Target/X86/X86MCInstLower.cpp

1151

NoAutoPaddingScope(MCStreamer &OS) : OS(OS) {

I recommend running command like git diff -U0 --no-color HEAD^ | clang-format-diff.py -i -p1 to format the patch.

1155–1157

keep the code in one line

1325

Remove blank here

[skan]

llvm/lib/Target/X86/X86MCInstLower.cpp

1149

The OldAllowAutoPadding is immutable indeed. I recommend declaring is as a const member and intializing it the initialization list.
Also, have your considered using the name DefaultAllowAutoPadding?

[reames]

Missed a review comment, now addressed.

[reames]

I've addressed the vast majority of style comments and have noted why I don't think the remainder are appropriate.

llvm/lib/Target/X86/X86MCInstLower.cpp

1155–1157

This is not idiomatic for non-accessors.

llvm/test/CodeGen/X86/align-branch-boundary-noautopadding.ll

2 ↗

(On Diff #236694)

On further reflection, I don't plan to add the textual assembly either. That exact case is already covered in the other test file.

[MaskRay]

llvm/test/CodeGen/X86/align-branch-boundary-noautopadding.ll

12 ↗

(On Diff #236848)

CHECK-NEXT: will be better.

llvm/include/llvm/MC/MCAsmBackend.h

virtual uint32_t
       generateCompactUnwindEncoding(ArrayRef<MCCFIInstruction>) const {
     return 0;
   }
 
   /// Check whether a given symbol has been flagged with MICROMIPS flag.
   virtual bool isMicroMips(const MCSymbol *Sym) const {
     return false;
   }
+
+  /// Given the target mask from a .push_align_branch_boundary directive,
+  /// return the string suitable an assembly file.
+  virtual std::string getAlignBranchBoundaryMaskStr(unsigned Mask) const {
+    llvm_unreachable("unsupported target for .push_align_branch_boundary?");
+    return "";
+  }
 };
 
 } // end namespace llvm
 
 #endif // LLVM_MC_MCASMBACKEND_H

llvm/include/llvm/MC/MCStreamer.h

class MCStreamer {
   /// The next unique ID to use when creating a WinCFI-related section (.pdata
   /// or .xdata). This ID ensures that we have a one-to-one mapping from
   /// code section to unwind info section, which MSVC's incremental linker
   /// requires.
   unsigned NextWinCFIID = 0;
 
   bool UseAssemblerInfoForParsing;
 
+  /// This is stack of align_branch_boundary directives.  Data is pair of
+  /// alignment, and target specific instruction kind mask.
+  SmallVector<std::pair<unsigned, unsigned>, 4> AlignBranchBoundaryStack;
+
 protected:
   MCStreamer(MCContext &Ctx);
 
   virtual void EmitCFIStartProcImpl(MCDwarfFrameInfo &Frame);
   virtual void EmitCFIEndProcImpl(MCDwarfFrameInfo &CurFrame);
 
   WinEH::FrameInfo *getCurrentWinFrameInfo() {
     return CurrentWinFrameInfo;
public:
   ///
   /// \param AlignToEnd - If true, the bundle-locked group will be aligned to
   ///                     the end of a bundle.
   virtual void EmitBundleLock(bool AlignToEnd);
 
   /// Ends a bundle-locked group.
   virtual void EmitBundleUnlock();
 
+  /// Enter a new .align_branch_boundary region which allows alignment for
+  /// performance of instruction described by the target specific Mask to be
+  /// aligned so as no to cross or end at an alignment boundary of 2^Align.
+  virtual void EmitPushAlignBranchBoundary(unsigned Align, unsigned Mask) {
+    AlignBranchBoundaryStack.push_back(std::make_pair(Align, Mask));
+  }
+
+  /// Ends a .align_branch_boundary region.
+  virtual void EmitPopAlignBranchBoundary() {
+    assert(!AlignBranchBoundaryStack.empty());
+    AlignBranchBoundaryStack.pop_back();
+  }
+
+  /// Return information on the current .align_branch_boundary region if any,
+  /// or None.
+  Optional<std::pair<unsigned, unsigned>>
+  getCurrentAlignBranchBoundary() const {
+    if (AlignBranchBoundaryStack.empty())
+      return None;
+    return AlignBranchBoundaryStack.back();
+  }
+
   /// If this file is backed by a assembly streamer, this dumps the
   /// specified string in the output .s file.  This capability is indicated by
   /// the hasRawTextSupport() predicate.  By default this aborts.
   void EmitRawText(const Twine &String);
 
   /// Streamer specific finalization.
   virtual void FinishImpl();
   /// Finish emission of machine code.

llvm/lib/MC/MCAsmStreamer.cpp

MCAsmStreamer(MCContext &Context, std::unique_ptr<formatted_raw_ostream> os,
             Context, std::move(asmbackend), std::move(emitter),
             (asmbackend) ? asmbackend->createObjectWriter(NullStream)
                          : nullptr)),
         CommentStream(CommentToEmit), IsVerboseAsm(isVerboseAsm),
         ShowInst(showInst), UseDwarfDirectory(useDwarfDirectory) {
     assert(InstPrinter);
     if (IsVerboseAsm)
         InstPrinter->setCommentStream(CommentStream);
   }
 
   MCAssembler &getAssembler() { return *Assembler; }
   MCAssembler *getAssemblerPtr() override { return nullptr; }
 
   inline void EmitEOL() {
     // Dump Explicit Comments here.
     emitExplicitComments();
     // If we don't have any comments, just emit a \n.
void emitCGProfileEntry(const MCSymbolRefExpr *From,
                           const MCSymbolRefExpr *To, uint64_t Count) override;
 
   void EmitInstruction(const MCInst &Inst, const MCSubtargetInfo &STI) override;
 
   void EmitBundleAlignMode(unsigned AlignPow2) override;
   void EmitBundleLock(bool AlignToEnd) override;
   void EmitBundleUnlock() override;
 
+  void EmitPushAlignBranchBoundary(unsigned Align, unsigned Mask) override;
+  void EmitPopAlignBranchBoundary() override;
+
   bool EmitRelocDirective(const MCExpr &Offset, StringRef Name,
                           const MCExpr *Expr, SMLoc Loc,
                           const MCSubtargetInfo &STI) override;
 
   void EmitAddrsig() override;
   void EmitAddrsigSym(const MCSymbol *Sym) override;
 
   /// If this file is backed by an assembly streamer, this dumps the specified
void MCAsmStreamer::EmitBundleLock(bool AlignToEnd) {
   EmitEOL();
 }
 
 void MCAsmStreamer::EmitBundleUnlock() {
   OS << "\t.bundle_unlock";
   EmitEOL();
 }
 
+void MCAsmStreamer::EmitPushAlignBranchBoundary(unsigned Align,
+                                                unsigned Mask) {
+  MCStreamer::EmitPushAlignBranchBoundary(Align, Mask);
+  OS << "\t.llvm_internal_push_align_branch_boundary ";
+  OS << Align << " ";
+  OS << getAssembler().getBackend().getAlignBranchBoundaryMaskStr(Mask);
+  EmitEOL();
+}
+
+void MCAsmStreamer::EmitPopAlignBranchBoundary() {
+  MCStreamer::EmitPopAlignBranchBoundary();
+  OS << "\t.llvm_internal_pop_align_branch_boundary";
+  EmitEOL();
+}
+
+
 bool MCAsmStreamer::EmitRelocDirective(const MCExpr &Offset, StringRef Name,
                                        const MCExpr *Expr, SMLoc,
                                        const MCSubtargetInfo &STI) {
   OS << "\t.reloc ";
   Offset.print(OS, MAI);
   OS << ", " << Name;
   if (Expr) {
     OS << ", ";

llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp

 #include "llvm/MC/MCParser/MCAsmParser.h"
 #include "llvm/MC/MCParser/MCParsedAsmOperand.h"
 #include "llvm/MC/MCParser/MCTargetAsmParser.h"
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/MC/MCSection.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/MC/MCSubtargetInfo.h"
 #include "llvm/MC/MCSymbol.h"
+#include "llvm/Support/CommandLine.h"
 #include "llvm/Support/SourceMgr.h"
 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <memory>
 
 using namespace llvm;
 
+
+cl::opt<bool> AllowLLVMInternalSyntax(
+    "parse-llvm-internal-testing-asm-directives", cl::init(false),
+    cl::ReallyHidden,
+    cl::desc(
+             "Support additional assembly directives which exist for LLVM's "
+             "internal testing purposes only.  These are not stable and no "
+             "compatibility will be provided in any form."));
+
 static bool checkScale(unsigned Scale, StringRef &ErrMsg) {
   if (Scale != 1 && Scale != 2 && Scale != 4 && Scale != 8) {
     ErrMsg = "scale factor in address must be 1, 2, 4 or 8";
     return true;
   }
   return false;
 }
 
private:
   /// SEH directives.
   bool parseSEHRegisterNumber(unsigned RegClassID, unsigned &RegNo);
   bool parseDirectiveSEHPushReg(SMLoc);
   bool parseDirectiveSEHSetFrame(SMLoc);
   bool parseDirectiveSEHSaveReg(SMLoc);
   bool parseDirectiveSEHSaveXMM(SMLoc);
   bool parseDirectiveSEHPushFrame(SMLoc);
 
+  /// .push_align_branch_boundary
+  bool parseDirectivePushAlignBranchBoundary(SMLoc);
+
   unsigned checkTargetMatchPredicate(MCInst &Inst) override;
 
   bool validateInstruction(MCInst &Inst, const OperandVector &Ops);
   bool processInstruction(MCInst &Inst, const OperandVector &Ops);
 
   /// Wrapper around MCStreamer::EmitInstruction(). Possibly adds
   /// instrumentation around Inst.
   void EmitInstruction(MCInst &Inst, OperandVector &Operands, MCStreamer &Out);
bool X86AsmParser::ParseDirective(AsmToken DirectiveID) {
   else if (IDVal == ".seh_setframe")
     return parseDirectiveSEHSetFrame(DirectiveID.getLoc());
   else if (IDVal == ".seh_savereg")
     return parseDirectiveSEHSaveReg(DirectiveID.getLoc());
   else if (IDVal == ".seh_savexmm")
     return parseDirectiveSEHSaveXMM(DirectiveID.getLoc());
   else if (IDVal == ".seh_pushframe")
     return parseDirectiveSEHPushFrame(DirectiveID.getLoc());
+  else if (AllowLLVMInternalSyntax &&
+           IDVal == ".llvm_internal_push_align_branch_boundary")
+    return parseDirectivePushAlignBranchBoundary(DirectiveID.getLoc());
+  else if (AllowLLVMInternalSyntax &&
+           IDVal == ".llvm_internal_pop_align_branch_boundary") {
+    if (None == getStreamer().getCurrentAlignBranchBoundary())
+      report_fatal_error("Mismatched .pop_align_branch_boundary");
+
+    if (getLexer().isNot(AsmToken::EndOfStatement))
+      return TokError("unexpected token in directive");
+    getParser().Lex();
+
+    getStreamer().EmitPopAlignBranchBoundary();
+    return false;
+  }
 
   return true;
 }
 
 /// parseDirectiveEven
 ///  ::= .even
 bool X86AsmParser::parseDirectiveEven(SMLoc L) {
   if (parseToken(AsmToken::EndOfStatement, "unexpected token in directive"))
bool X86AsmParser::parseDirectiveSEHPushFrame(SMLoc Loc) {
   if (getLexer().isNot(AsmToken::EndOfStatement))
     return TokError("unexpected token in directive");
 
   getParser().Lex();
   getStreamer().EmitWinCFIPushFrame(Code, Loc);
   return false;
 }
 
+bool X86AsmParser::parseDirectivePushAlignBranchBoundary(SMLoc Loc) {
+  int64_t BoundaryLog2;
+  if (getParser().parseAbsoluteExpression(BoundaryLog2))
+    return TokError("expected power-of-2 aligment expression");
+
+  unsigned Mask = 0;
+  if (getLexer().isNot(AsmToken::EndOfStatement)) {
+    do {
+      StringRef Name;
+      if (getParser().parseIdentifier(Name))
+        return TokError("Missing instruction type");
+      unsigned MaskBit = StringSwitch<int>(Name)
+        .Case("fused", X86::AlignBranchFused)
+        .Case("jcc", X86::AlignBranchJcc)
+        .Case("jmp", X86::AlignBranchJmp)
+        .Case("call", X86::AlignBranchCall)
+        .Case("ret", X86::AlignBranchRet)
+        .Case("indirect", X86::AlignBranchIndirect)
+        .Default(-1);
+      if (MaskBit == -1U)
+        return TokError("unrecognized instruction type in directive");
+      Mask |= MaskBit;
+    } while (parseOptionalToken(AsmToken::Comma));
+  }
+  if (getLexer().isNot(AsmToken::EndOfStatement))
+    return TokError("unexpected token in directive");
+  getParser().Lex();
+  getStreamer().EmitPushAlignBranchBoundary(BoundaryLog2, Mask);
+  return false;
+}
+
 // Force static initialization.
 extern "C" void LLVMInitializeX86AsmParser() {
   RegisterMCAsmParser<X86AsmParser> X(getTheX86_32Target());
   RegisterMCAsmParser<X86AsmParser> Y(getTheX86_64Target());
 }
 
 #define GET_REGISTER_MATCHER
 #define GET_MATCHER_IMPLEMENTATION
 #define GET_SUBTARGET_FEATURE_NAME
 #include "X86GenAsmMatcher.inc"

llvm/lib/Target/X86/MCTargetDesc/X86AsmBackend.cpp

 //===-- X86AsmBackend.cpp - X86 Assembler Backend -------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "MCTargetDesc/X86BaseInfo.h"
 #include "MCTargetDesc/X86FixupKinds.h"
+#include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/BinaryFormat/ELF.h"
 #include "llvm/BinaryFormat/MachO.h"
 #include "llvm/MC/MCAsmBackend.h"
 #include "llvm/MC/MCAssembler.h"
 #include "llvm/MC/MCContext.h"
 #include "llvm/MC/MCDwarf.h"
 #include "llvm/MC/MCELFObjectWriter.h"
public:
   bool fixupNeedsRelaxation(const MCFixup &Fixup, uint64_t Value,
                             const MCRelaxableFragment *DF,
                             const MCAsmLayout &Layout) const override;
 
   void relaxInstruction(const MCInst &Inst, const MCSubtargetInfo &STI,
                         MCInst &Res) const override;
 
   bool writeNopData(raw_ostream &OS, uint64_t Count) const override;
+
+  std::string getAlignBranchBoundaryMaskStr(unsigned Mask) const override;
 };
 } // end anonymous namespace
 
 static unsigned getRelaxedOpcodeBranch(const MCInst &Inst, bool is16BitMode) {
   unsigned Op = Inst.getOpcode();
   switch (Op) {
   default:
     return Op;
do {
     if (Rest != 0)
       OS.write(Nops[Rest - 1], Rest);
     Count -= ThisNopLength;
   } while (Count != 0);
 
   return true;
 }
 
+std::string X86AsmBackend::getAlignBranchBoundaryMaskStr(unsigned Mask) const {
+  SmallVector<std::string, 8> Strings;
+  if (Mask & X86::AlignBranchFused)
+    Strings.push_back("fused");
+  if (Mask & X86::AlignBranchJcc)
+    Strings.push_back("jcc");
+  if (Mask & X86::AlignBranchJmp)
+    Strings.push_back("jmp");
+  if (Mask & X86::AlignBranchCall)
+    Strings.push_back("call");
+  if (Mask & X86::AlignBranchRet)
+    Strings.push_back("ret");
+  if (Mask & X86::AlignBranchIndirect)
+    Strings.push_back("indirect");
+
+  return join(Strings, ", ");
+}
+
 /* *** */
 
 namespace {
 
 class ELFX86AsmBackend : public X86AsmBackend {
 public:
   uint8_t OSABI;
   ELFX86AsmBackend(const Target &T, uint8_t OSABI, const MCSubtargetInfo &STI)

llvm/lib/Target/X86/MCTargetDesc/X86BaseInfo.h

case X86::FirstMacroFusionInstKind::AddSub:
              SecondKind == X86::SecondMacroFusionInstKind::ELG;
     case X86::FirstMacroFusionInstKind::IncDec:
       return SecondKind == X86::SecondMacroFusionInstKind::ELG;
     case X86::FirstMacroFusionInstKind::Invalid:
       return false;
     }
     llvm_unreachable("unknown fusion type");
   }
+  
+  /// Defines the value used for the target specific mask in a scope for
+  /// align_branch_boundary
+  enum AlignBranchBoundaryKinds : uint8_t {
+    AlignBranchNone = 0,
+    AlignBranchFused = 1U << 0,
+    AlignBranchJcc = 1U << 1,
+    AlignBranchJmp = 1U << 2,
+    AlignBranchCall = 1U << 3,
+    AlignBranchRet = 1U << 4,
+    AlignBranchIndirect = 1U << 5
+  };
+
 } // end namespace X86;
 
 /// X86II - This namespace holds all of the target specific flags that
 /// instruction info tracks.
 ///
 namespace X86II {
   /// Target Operand Flag enum.
   enum TOF {

llvm/lib/Target/X86/X86AsmPrinter.cpp

if (Subtarget->isTargetCOFF()) {
     OutStreamer->BeginCOFFSymbolDef(CurrentFnSym);
     OutStreamer->EmitCOFFSymbolStorageClass(
         Local ? COFF::IMAGE_SYM_CLASS_STATIC : COFF::IMAGE_SYM_CLASS_EXTERNAL);
     OutStreamer->EmitCOFFSymbolType(COFF::IMAGE_SYM_DTYPE_FUNCTION
                                                << COFF::SCT_COMPLEX_TYPE_SHIFT);
     OutStreamer->EndCOFFSymbolDef();
   }
 
+  // Some Intel processors require additional alignment padding to mitigate an
+  // performance issue introduced  microcode update fix for a correctness issue
+  // which negative impacted uop caching of branches which cross a 32 byte
+  // boundary.  For the moment, CLFLUSHOPT is being used a proxy for which
+  // machines need the mitigation since this was also introduced in the
+  // affected generation (skylake), but this needs split out into it's own flag
+  // (FIXME BEFORE COMMIT). 
+  const bool NeedsJCCAlignMitigation = Subtarget->hasCLFLUSHOPT();
+  if (NeedsJCCAlignMitigation) {
+    const uint8_t DefaultAligned =
+      (X86::AlignBranchFused | X86::AlignBranchJcc | X86::AlignBranchJmp);
+    OutStreamer->EmitPushAlignBranchBoundary(5, DefaultAligned);
+  }
+
   // Emit the rest of the function body.
   EmitFunctionBody();
 
+  if (NeedsJCCAlignMitigation)
+    OutStreamer->EmitPopAlignBranchBoundary();
+
   // Emit the XRay table for this function.
   emitXRayTable();
 
   EmitFPOData = false;
 
   // We didn't modify anything.
   return false;
 }

llvm/lib/Target/X86/X86MCInstLower.cpp

static void EmitNops(MCStreamer &OS, unsigned NumBytes, bool Is64Bit,
   unsigned NopsToEmit = NumBytes;
   (void)NopsToEmit;
   while (NumBytes) {
     NumBytes -= EmitNop(OS, NumBytes, Is64Bit, STI);
     assert(NopsToEmit >= NumBytes && "Emitted more than I asked for!");
   }
 }
 
+/// A RAII helper which defines a region of instructions which can't have
+/// padding added between them for correctness.  This is a nop unless we're in
+/// a branch_boundary_align region, in which case it pushes a new scope with
+/// all padding kinds disabled, and then pops it at end of scope.
+struct NoAutoPaddingScope {
+  MCStreamer &OS;
+  bool EmitPop = false;
+  NoAutoPaddingScope(MCStreamer &OS)
+    : OS(OS) {
+    if (auto AlignInfo = OS.getCurrentAlignBranchBoundary()) {
+      // If active, push a new scope with all padding disabled
+      OS.EmitPushAlignBranchBoundary(AlignInfo->first, 0);
+      EmitPop = true;
+    }
+  }
+  ~NoAutoPaddingScope() {
+    if (EmitPop)
+      OS.EmitPopAlignBranchBoundary();
+  }
+};
+
 void X86AsmPrinter::LowerSTATEPOINT(const MachineInstr &MI,
                                     X86MCInstLower &MCIL) {
   assert(Subtarget->is64Bit() && "Statepoint currently only supports X86-64");
 
+  NoAutoPaddingScope NoPadScope(*OutStreamer);
+
   StatepointOpers SOpers(&MI);
   if (unsigned PatchBytes = SOpers.getNumPatchBytes()) {
     EmitNops(*OutStreamer, PatchBytes, Subtarget->is64Bit(),
              getSubtargetInfo());
   } else {
     // Lower call target and choose correct opcode
     const MachineOperand &CallTarget = SOpers.getCallTarget();
     MCOperand CallTargetMCOp;
void X86AsmPrinter::LowerSTATEPOINT(const MachineInstr &MI,
   SM.recordStatepoint(*MILabel, MI);
 }
 
 void X86AsmPrinter::LowerFAULTING_OP(const MachineInstr &FaultingMI,
                                      X86MCInstLower &MCIL) {
   // FAULTING_LOAD_OP <def>, <faltinf type>, <MBB handler>,
   //                  <opcode>, <operands>
 
+  NoAutoPaddingScope NoPadScope(*OutStreamer);
+
   Register DefRegister = FaultingMI.getOperand(0).getReg();
   FaultMaps::FaultKind FK =
       static_cast<FaultMaps::FaultKind>(FaultingMI.getOperand(1).getImm());
   MCSymbol *HandlerLabel = FaultingMI.getOperand(2).getMBB()->getSymbol();
   unsigned Opcode = FaultingMI.getOperand(3).getImm();
   unsigned OperandsBeginIdx = 4;
 
   auto &Ctx = OutStreamer->getContext();
void X86AsmPrinter::LowerFENTRY_CALL(const MachineInstr &MI,
   EmitAndCountInstruction(
       MCInstBuilder(Is64Bits ? X86::CALL64pcrel32 : X86::CALLpcrel32)
           .addExpr(Op));
 }
 
 void X86AsmPrinter::LowerPATCHABLE_OP(const MachineInstr &MI,
                                       X86MCInstLower &MCIL) {
   // PATCHABLE_OP minsize, opcode, operands
   
+  NoAutoPaddingScope NoPadScope(*OutStreamer);
+
   unsigned MinSize = MI.getOperand(0).getImm();
   unsigned Opcode = MI.getOperand(1).getImm();
 
   MCInst MCI;
   MCI.setOpcode(Opcode);
   for (auto &MO : make_range(MI.operands_begin() + 2, MI.operands_end()))
     if (auto MaybeOperand = MCIL.LowerMachineOperand(&MI, MO))
       MCI.addOperand(MaybeOperand.getValue());
void X86AsmPrinter::LowerPATCHABLE_OP(const MachineInstr &MI,
   OutStreamer->EmitInstruction(MCI, getSubtargetInfo());
 }
 
 // Lower a stackmap of the form:
 // <id>, <shadowBytes>, ...
 void X86AsmPrinter::LowerSTACKMAP(const MachineInstr &MI) {
   SMShadowTracker.emitShadowPadding(*OutStreamer, getSubtargetInfo());
 
+  NoAutoPaddingScope NoPadScope(*OutStreamer);  
+  
   auto &Ctx = OutStreamer->getContext();
   MCSymbol *MILabel = Ctx.createTempSymbol();
   OutStreamer->EmitLabel(MILabel);
 
   SM.recordStackMap(*MILabel, MI);
   unsigned NumShadowBytes = MI.getOperand(1).getImm();
   SMShadowTracker.reset(NumShadowBytes);
 }
 
 // Lower a patchpoint of the form:
 // [<def>], <id>, <numBytes>, <target>, <numArgs>, <cc>, ...
 void X86AsmPrinter::LowerPATCHPOINT(const MachineInstr &MI,
                                     X86MCInstLower &MCIL) {
   assert(Subtarget->is64Bit() && "Patchpoint currently only supports X86-64");
 
   SMShadowTracker.emitShadowPadding(*OutStreamer, getSubtargetInfo());
 
+  NoAutoPaddingScope NoPadScope(*OutStreamer);
+
   auto &Ctx = OutStreamer->getContext();
   MCSymbol *MILabel = Ctx.createTempSymbol();
   OutStreamer->EmitLabel(MILabel);
   SM.recordPatchPoint(*MILabel, MI);
 
   PatchPointOpers opers(&MI);
   unsigned ScratchIdx = opers.getNextScratchIdx();
   unsigned EncodedBytes = 0;
void X86AsmPrinter::LowerPATCHPOINT(const MachineInstr &MI,
   EmitNops(*OutStreamer, NumBytes - EncodedBytes, Subtarget->is64Bit(),
            getSubtargetInfo());
 }
 
 void X86AsmPrinter::LowerPATCHABLE_EVENT_CALL(const MachineInstr &MI,
                                               X86MCInstLower &MCIL) {
   assert(Subtarget->is64Bit() && "XRay custom events only supports X86-64");
 
+  NoAutoPaddingScope NoPadScope(*OutStreamer);
+
   // We want to emit the following pattern, which follows the x86 calling
   // convention to prepare for the trampoline call to be patched in.
   //
   //   .p2align 1, ...
   // .Lxray_event_sled_N:
   //   jmp +N                        // jump across the instrumentation sled
   //   ...                           // set up arguments in register
   //   callq __xray_CustomEvent@plt  // force dependency to symbol
void X86AsmPrinter::LowerPATCHABLE_EVENT_CALL(const MachineInstr &MI,
   // using.
   recordSled(CurSled, MI, SledKind::CUSTOM_EVENT, 1);
 }
 
 void X86AsmPrinter::LowerPATCHABLE_TYPED_EVENT_CALL(const MachineInstr &MI,
                                                     X86MCInstLower &MCIL) {
   assert(Subtarget->is64Bit() && "XRay typed events only supports X86-64");
 
+  NoAutoPaddingScope NoPadScope(*OutStreamer);
+
   // We want to emit the following pattern, which follows the x86 calling
   // convention to prepare for the trampoline call to be patched in.
   //
   //   .p2align 1, ...
   // .Lxray_event_sled_N:
   //   jmp +N                        // jump across the instrumentation sled
   //   ...                           // set up arguments in register
   //   callq __xray_TypedEvent@plt  // force dependency to symbol
void X86AsmPrinter::LowerPATCHABLE_TYPED_EVENT_CALL(const MachineInstr &MI,
   OutStreamer->AddComment("xray typed event end.");
 
   // Record the sled version.
   recordSled(CurSled, MI, SledKind::TYPED_EVENT, 0);
 }
 
 void X86AsmPrinter::LowerPATCHABLE_FUNCTION_ENTER(const MachineInstr &MI,
                                                   X86MCInstLower &MCIL) {
+
+  NoAutoPaddingScope NoPadScope(*OutStreamer);
+  
   // We want to emit the following pattern:
   //
   //   .p2align 1, ...
   // .Lxray_sled_N:
   //   jmp .tmpN
   //   # 9 bytes worth of noops
   //
   // We need the 9 bytes because at runtime, we'd be patching over the full 11
void X86AsmPrinter::LowerPATCHABLE_FUNCTION_ENTER(const MachineInstr &MI,
   // FIXME: Find another less hacky way do force the relative jump.
   OutStreamer->EmitBytes("\xeb\x09");
   EmitNops(*OutStreamer, 9, Subtarget->is64Bit(), getSubtargetInfo());
   recordSled(CurSled, MI, SledKind::FUNCTION_ENTER);
 }
 
 void X86AsmPrinter::LowerPATCHABLE_RET(const MachineInstr &MI,
                                        X86MCInstLower &MCIL) {
+  NoAutoPaddingScope NoPadScope(*OutStreamer);
+
   // Since PATCHABLE_RET takes the opcode of the return statement as an
   // argument, we use that to emit the correct form of the RET that we want.
   // i.e. when we see this:
   //
   //   PATCHABLE_RET X86::RET ...
   //
   // We should emit the RET followed by sleds.
   //
if (auto MaybeOperand = MCIL.LowerMachineOperand(&MI, MO))
       Ret.addOperand(MaybeOperand.getValue());
   OutStreamer->EmitInstruction(Ret, getSubtargetInfo());
   EmitNops(*OutStreamer, 10, Subtarget->is64Bit(), getSubtargetInfo());
   recordSled(CurSled, MI, SledKind::FUNCTION_EXIT);
 }
 
 void X86AsmPrinter::LowerPATCHABLE_TAIL_CALL(const MachineInstr &MI,
                                              X86MCInstLower &MCIL) {
+  NoAutoPaddingScope NoPadScope(*OutStreamer);
+
   // Like PATCHABLE_RET, we have the actual instruction in the operands to this
   // instruction so we lower that particular instruction and its operands.
   // Unlike PATCHABLE_RET though, we put the sled before the JMP, much like how
   // we do it for PATCHABLE_FUNCTION_ENTER. The sled should be very similar to
   // the PATCHABLE_FUNCTION_ENTER case, followed by the lowering of the actual
   // tail call much like how we have it in PATCHABLE_RET.
   auto CurSled = OutContext.createTempSymbol("xray_sled_", true);
   OutStreamer->EmitCodeAlignment(2);

llvm/test/CodeGen/X86/implicit-null-check-noautopad.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -verify-machineinstrs -O3 -mtriple=x86_64-apple-macosx -enable-implicit-null-checks -mcpu=skylake < %s | FileCheck %s
+
+; If we have autopadding enabled, make sure the label isn't separated from
+; the mov.
+
+define i32 @test(i32* %x) {
+; CHECK-LABEL: test:
+; CHECK:       ## %bb.0: ## %entry
+; CHECK-NEXT:    .llvm_internal_push_align_branch_boundary 5
+; CHECK-NEXT:  Ltmp0:
+; CHECK-NEXT:    movl (%rdi), %eax ## on-fault: LBB0_1
+; CHECK-NEXT:    .llvm_internal_pop_align_branch_boundary
+; CHECK-NEXT:  ## %bb.2: ## %not_null
+; CHECK-NEXT:    retq
+; CHECK-NEXT:  LBB0_1: ## %is_null
+; CHECK-NEXT:    movl $42, %eax
+; CHECK-NEXT:    retq
+
+ entry:
+  %c = icmp eq i32* %x, null
+  br i1 %c, label %is_null, label %not_null, !make.implicit !0
+
+ is_null:
+  ret i32 42
+
+ not_null:
+  %t = load atomic i32, i32* %x unordered, align 4
+  ret i32 %t
+}
+
+!0 = !{}

llvm/test/CodeGen/X86/statepoint-no-realign-stack.ll

 ; CHECK-NEXT:    .cfi_offset %rbp, -16
 ; CHECK-NEXT:    movq %rsp, %rbp
 ; CHECK-NEXT:    .cfi_def_cfa_register %rbp
 ; CHECK-NEXT:    andq $-32, %rsp
 ; CHECK-NEXT:    subq $64, %rsp
 ; CHECK-NEXT:    vmovaps (%rdi), %ymm0
 ; CHECK-NEXT:    vmovaps %ymm0, (%rsp)
 ; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    .llvm_internal_push_align_branch_boundary 5
 ; CHECK-NEXT:    callq foo
 ; CHECK-NEXT:  .Ltmp0:
+; CHECK-NEXT:    .llvm_internal_pop_align_branch_boundary
 ; CHECK-NEXT:    movq %rbp, %rsp
 ; CHECK-NEXT:    popq %rbp
 ; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
 ; CHECK-NEXT:    retq
   %val = load <8 x i32>, <8 x i32>* %p, align 32
   call void @foo() ["deopt" (<8 x i32> %val)]
   ret void
 }
 
 define void @no_realign(<8 x i32>* %p) "no-realign-stack" {
 ; CHECK-LABEL: no_realign:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    subq $40, %rsp
 ; CHECK-NEXT:    .cfi_def_cfa_offset 48
 ; CHECK-NEXT:    vmovaps (%rdi), %ymm0
 ; CHECK-NEXT:    vmovups %ymm0, (%rsp)
 ; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    .llvm_internal_push_align_branch_boundary 5
 ; CHECK-NEXT:    callq foo
 ; CHECK-NEXT:  .Ltmp1:
+; CHECK-NEXT:    .llvm_internal_pop_align_branch_boundary
 ; CHECK-NEXT:    addq $40, %rsp
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
   %val = load <8 x i32>, <8 x i32>* %p, align 32
   call void @foo() ["deopt" (<8 x i32> %val)]
   ret void
 }
 
 ;; Next batch are similiar to the above, but require a reload of the
 ;; spilled value as well.
 
 define <4 x i8 addrspace(1)*> @spillfill_can_realign(<4 x i8 addrspace(1)*> %obj) gc "statepoint-example" {
 ; CHECK-LABEL: spillfill_can_realign:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushq %rbp
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    .cfi_offset %rbp, -16
 ; CHECK-NEXT:    movq %rsp, %rbp
 ; CHECK-NEXT:    .cfi_def_cfa_register %rbp
 ; CHECK-NEXT:    andq $-32, %rsp
 ; CHECK-NEXT:    subq $64, %rsp
 ; CHECK-NEXT:    vmovaps %ymm0, (%rsp)
 ; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    .llvm_internal_push_align_branch_boundary 5
 ; CHECK-NEXT:    callq do_safepoint
 ; CHECK-NEXT:  .Ltmp2:
+; CHECK-NEXT:    .llvm_internal_pop_align_branch_boundary
 ; CHECK-NEXT:    vmovaps (%rsp), %ymm0
 ; CHECK-NEXT:    movq %rbp, %rsp
 ; CHECK-NEXT:    popq %rbp
 ; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
 ; CHECK-NEXT:    retq
 entry:
   %safepoint_token = call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @do_safepoint, i32 0, i32 0, i32 0, i32 0, <4 x i8 addrspace(1)*> %obj)
   %obj.relocated = call coldcc <4 x i8 addrspace(1)*> @llvm.experimental.gc.relocate.v4p1i8(token %safepoint_token, i32 7, i32 7) ; (%obj, %obj)
   ret <4 x i8 addrspace(1)*> %obj.relocated
 }
 
 define <4 x i8 addrspace(1)*> @spillfill_no_realign(<4 x i8 addrspace(1)*> %obj) "no-realign-stack" gc "statepoint-example" {
 ; CHECK-LABEL: spillfill_no_realign:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    subq $40, %rsp
 ; CHECK-NEXT:    .cfi_def_cfa_offset 48
 ; CHECK-NEXT:    vmovups %ymm0, (%rsp)
 ; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    .llvm_internal_push_align_branch_boundary 5
 ; CHECK-NEXT:    callq do_safepoint
 ; CHECK-NEXT:  .Ltmp3:
+; CHECK-NEXT:    .llvm_internal_pop_align_branch_boundary
 ; CHECK-NEXT:    vmovups (%rsp), %ymm0
 ; CHECK-NEXT:    addq $40, %rsp
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
 entry:
   %safepoint_token = call token (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* @do_safepoint, i32 0, i32 0, i32 0, i32 0, <4 x i8 addrspace(1)*> %obj)
   %obj.relocated = call coldcc <4 x i8 addrspace(1)*> @llvm.experimental.gc.relocate.v4p1i8(token %safepoint_token, i32 7, i32 7) ; (%obj, %obj)
   ret <4 x i8 addrspace(1)*> %obj.relocated
 }
 
 declare void @do_safepoint()
 
 declare token @llvm.experimental.gc.statepoint.p0f_isVoidf(i64, i32, void ()*, i32, i32, ...)
 declare i8 addrspace(1)* @llvm.experimental.gc.relocate.p1i8(token, i32, i32)
 declare <4 x i8 addrspace(1)*> @llvm.experimental.gc.relocate.v4p1i8(token, i32, i32)

llvm/test/MC/X86/align-branch-boundary.s

+  # RUN: llvm-mc -triple x86_64-pc-linux-gnu -parse-llvm-internal-testing-asm-directives %s | FileCheck %s
+
+  # basic sanity check for round trip serialization of the proposed
+  # assembler extension. Many more tests needed of course
+
+  # CHECK: test1:
+  # CHECK-NEXT: .llvm_internal_push_align_branch_boundary 7 
+  # CHECK-NEXT: .llvm_internal_push_align_branch_boundary 5 jmp, ret, indirect
+  # CHECK-NEXT: .llvm_internal_push_align_branch_boundary 5 fused, jcc, call
+  # CHECK-NEXT: callq bar
+  # CHECK-NEXT: .llvm_internal_pop_align_branch_boundary
+  # CHECK-NEXT: .llvm_internal_pop_align_branch_boundary
+  # CHECK-NEXT: .llvm_internal_pop_align_branch_boundary
+  .text
+  .globl  test1
+  .p2align  5
+test1:
+  .llvm_internal_push_align_branch_boundary 7 #no instruction type
+  .llvm_internal_push_align_branch_boundary 5 jmp, indirect, ret
+  .llvm_internal_push_align_branch_boundary 5 jcc, fused, call
+  callq bar
+  .llvm_internal_pop_align_branch_boundary
+  .llvm_internal_pop_align_branch_boundary
+  .llvm_internal_pop_align_branch_boundary
+bar:
+  retq
+
+  # a far target (4 byte imm)
+  .section "unknown"
+  .type   baz,@function
+baz:
+  retq