[MC] Write padding into fragments when -mc-relax-all flag is used

[phosek]

Summary

When instruction bundling is enabled and the -mc-relax-all flag is
set, we can write bundle padding directly into fragments and avoid
creating large number of fragments significantly reducing LLVM MC
memory usage.

Fixes https://code.google.com/p/nativeclient/issues/detail?id=4063

[eliben]

Could you first update http://www.chromium.org/nativeclient/pnacl/aligned-bundling-support-in-llvm to explain the proposed scheme in more general terms?

[eliben]

include/llvm/MC/MCAsmLayout.h

112 ↗

(On Diff #21251)

Is there a good reason to keep this a member and not just a standalone function?

include/llvm/MC/MCAssembler.h

1245 ↗

(On Diff #21251)

Here and elsewhere - please document all the new functions/methods you add - explaining their arguments, return values, etc.

include/llvm/MC/MCObjectStreamer.h

86 ↗

(On Diff #21251)

What is FOffset for?

lib/MC/MCAssembler.cpp

633 ↗

(On Diff #21251)

Does this comment need updating?

tools/pnacl-llc/pnacl-llc.cpp

706 ↗

(On Diff #21251)

This file doesn't see to belong here

[phosek]

Addressing code review comments.

[phosek]

I have added the description of the proposed scheme to http://www.chromium.org/nativeclient/pnacl/aligned-bundling-support-in-llvm as requested.

include/llvm/MC/MCAsmLayout.h

112 ↗

(On Diff #21251)

Refactored as a standalone function.

include/llvm/MC/MCAssembler.h

1245 ↗

(On Diff #21251)

Done for all the new functions/methods.

include/llvm/MC/MCObjectStreamer.h

86 ↗

(On Diff #21251)

I have update the comment to address that.

lib/MC/MCAssembler.cpp

633 ↗

(On Diff #21251)

I have update the comment to mention the optimization used when relax all flag is used.

tools/pnacl-llc/pnacl-llc.cpp

706 ↗

(On Diff #21251)

Right, this is a leftover testing artifact; removed.

[eliben]

include/llvm/MC/MCAssembler.h

1252 ↗

(On Diff #21532)

Please place all method comments consistently on the method declaration, rather than definition

1261 ↗

(On Diff #21532)

Describe FOffest and FSize

include/llvm/MC/MCELFStreamer.h

117 ↗

(On Diff #21532)

Describe this new data structure

lib/MC/MCAssembler.cpp

657 ↗

(On Diff #21532)

writing

725 ↗

(On Diff #21532)

So this FIXME is potentially addressed?

lib/MC/MCELFStreamer.cpp

496 ↗

(On Diff #21532)

Each clause here had a comment explaining what's going on - please preserve that

568 ↗

(On Diff #21532)

Add a comment above this block explaining what it's doing

589 ↗

(On Diff #21532)

Add a comment above this block explaining what it's doing

[phosek]

Addressing code review comments.

[eliben]

LGTM

[mseaborn]

llvm/trunk/lib/MC/MCObjectStreamer.cpp

148

What is the reason for checking isBundlingEnabled() and getRelaxAll() here?

Could this explain why references to data sections are coming out wrong? (See the bug described here: https://chromiumcodereview.appspot.com/1129013004/#msg5)

You are calling flushPendingLabels() from mergeFragment(), but that doesn't get called if we are processing data sections. (mergeFragment() is called from EmitInstToData().)

When we are processing data directives such as ".long" or ".asciz", what would cause labels to get their offsets set (via setOffset())?

llvm/trunk/include/llvm/MC/MCAsmLayout.h

private:
 
   /// \brief Make sure that the layout for the given fragment is valid, lazily
   /// computing it if necessary.
   void ensureValid(const MCFragment *F) const;
 
   /// \brief Is the layout for this fragment valid?
   bool isFragmentValid(const MCFragment *F) const;
 
-  /// \brief Compute the amount of padding required before this fragment to
-  /// obey bundling restrictions.
-  uint64_t computeBundlePadding(const MCFragment *F,
-                                uint64_t FOffset, uint64_t FSize);
-
 public:
   MCAsmLayout(MCAssembler &Assembler);
 
   /// Get the assembler object this is a layout for.
   MCAssembler &getAssembler() const { return Assembler; }
 
   /// \brief Invalidate the fragments starting with F because it has been
   /// resized. The fragment's size should have already been updated, but

llvm/trunk/include/llvm/MC/MCAssembler.h

public:
   }
 
   void addFileName(StringRef FileName) {
     if (std::find(file_names_begin(), file_names_end(), FileName) ==
         file_names_end())
       FileNames.push_back(FileName);
   }
 
+  /// \brief Write the necessary bundle padding to the given object writer.
+  /// Expects a fragment \p F containing instructions and its size \p FSize.
+  void writeFragmentPadding(const MCFragment &F, uint64_t FSize,
+                            MCObjectWriter *OW) const;
+
   /// @}
 
   void dump();
 };
 
+/// \brief Compute the amount of padding required before the fragment \p F to
+/// obey bundling restrictions, where \p FOffset is the fragment's offset in
+/// its section and \p FSize is the fragment's size.
+uint64_t computeBundlePadding(const MCAssembler &Assembler,
+                              const MCFragment *F,
+                              uint64_t FOffset, uint64_t FSize);
+
 } // end namespace llvm
 
 #endif

llvm/trunk/include/llvm/MC/MCELFStreamer.h

public:
   void EmitBundleUnlock() override;
 
 private:
   void EmitInstToFragment(const MCInst &Inst, const MCSubtargetInfo &) override;
   void EmitInstToData(const MCInst &Inst, const MCSubtargetInfo &) override;
 
   void fixSymbolsInTLSFixups(const MCExpr *expr);
 
+  /// \brief Merge the content of the fragment \p EF into the fragment \p DF.
+  void mergeFragment(MCDataFragment *, MCEncodedFragmentWithFixups *);
+
   bool SeenIdent;
 
   struct LocalCommon {
     MCSymbolData *SD;
     uint64_t Size;
     unsigned ByteAlignment;
   };
 
   std::vector<LocalCommon> LocalCommons;
 
   SmallPtrSet<MCSymbol *, 16> BindingExplicitlySet;
+
+  /// BundleGroups - The stack of fragments holding the bundle-locked
+  /// instructions.
+  llvm::SmallVector<MCDataFragment *, 4> BundleGroups;
 };
 
 MCELFStreamer *createARMELFStreamer(MCContext &Context, MCAsmBackend &TAB,
                                     raw_ostream &OS, MCCodeEmitter *Emitter,
                                     bool RelaxAll, bool IsThumb);
 
 } // end namespace llvm
 
 #endif

llvm/trunk/include/llvm/MC/MCObjectStreamer.h

class MCObjectStreamer : public MCStreamer {
   bool EmitEHFrame;
   bool EmitDebugFrame;
   SmallVector<MCSymbolData *, 2> PendingLabels;
 
   virtual void EmitInstToData(const MCInst &Inst, const MCSubtargetInfo&) = 0;
   void EmitCFIStartProcImpl(MCDwarfFrameInfo &Frame) override;
   void EmitCFIEndProcImpl(MCDwarfFrameInfo &Frame) override;
 
-  // If any labels have been emitted but not assigned fragments, ensure that
-  // they get assigned, either to F if possible or to a new data fragment.
-  void flushPendingLabels(MCFragment *F);
-
 protected:
   MCObjectStreamer(MCContext &Context, MCAsmBackend &TAB, raw_ostream &OS,
                    MCCodeEmitter *Emitter);
   ~MCObjectStreamer() override;
 
 public:
   /// state management
   void reset() override;
protected:
   }
 
   /// Get a data fragment to write into, creating a new one if the current
   /// fragment is not a data fragment.
   MCDataFragment *getOrCreateDataFragment();
 
   bool changeSectionImpl(const MCSection *Section, const MCExpr *Subsection);
 
+  /// If any labels have been emitted but not assigned fragments, ensure that
+  /// they get assigned, either to F if possible or to a new data fragment.
+  /// Optionally, it is also possible to provide an offset \p FOffset, which
+  /// will be used as a symbol offset within the fragment.
+  void flushPendingLabels(MCFragment *F, uint64_t FOffset = 0);
+
 public:
   void visitUsedSymbol(const MCSymbol &Sym) override;
 
   MCAssembler &getAssembler() { return *Assembler; }
 
   /// @name MCStreamer Interface
   /// @{
 

llvm/trunk/lib/MC/MCAssembler.cpp

uint64_t MCAsmLayout::getSectionFileSize(const MCSectionData *SD) const {
   // Virtual sections have no file size.
   if (SD->getSection().isVirtualSection())
     return 0;
 
   // Otherwise, the file size is the same as the address space size.
   return getSectionAddressSize(SD);
 }
 
-uint64_t MCAsmLayout::computeBundlePadding(const MCFragment *F,
-                                           uint64_t FOffset, uint64_t FSize) {
+uint64_t llvm::computeBundlePadding(const MCAssembler &Assembler,
+                                    const MCFragment *F,
   uint64_t BundleSize = Assembler.getBundleAlignSize();
   assert(BundleSize > 0 &&
          "computeBundlePadding should only be called if bundling is enabled");
   uint64_t BundleMask = BundleSize - 1;
   uint64_t OffsetInBundle = FOffset & BundleMask;
   uint64_t EndOfFragment = OffsetInBundle + FSize;
 
   // There are two kinds of bundling restrictions:
MCSectionData::getSubsectionInsertionPoint(unsigned Subsection) {
   if (!ExactMatch && Subsection != 0) {
     // The GNU as documentation claims that subsections have an alignment of 4,
     // although this appears not to be the case.
     MCFragment *F = new MCDataFragment();
     SubsectionFragmentMap.insert(MI, std::make_pair(Subsection, F));
     getFragmentList().insert(IP, F);
     F->setParent(this);
   }
+
   return IP;
 }
 
 void MCSectionData::setBundleLockState(BundleLockStateType NewState) {
   if (NewState == NotBundleLocked) {
     if (BundleLockNestingDepth == 0) {
       report_fatal_error("Mismatched bundle_lock/unlock directives");
     }
void MCAsmLayout::layoutFragment(MCFragment *F) {
   // -------------------------------------
   //                    ^
   //                    |
   //                    F->Offset
   //
   // The fragment's offset will point to after the padding, and its computed
   // size won't include the padding.
   //
-  if (Assembler.isBundlingEnabled() && F->hasInstructions()) {
+  // When the -mc-relax-all flag is used, we optimize bundling by writting the
+  // bundle padding directly into fragments when the instructions are emitted
+  // inside the streamer.
+  //
+  if (Assembler.isBundlingEnabled() && !Assembler.getRelaxAll() &&
+      F->hasInstructions()) {
     assert(isa<MCEncodedFragment>(F) &&
            "Only MCEncodedFragment implementations have instructions");
     uint64_t FSize = Assembler.computeFragmentSize(*this, *F);
 
     if (FSize > Assembler.getBundleAlignSize())
       report_fatal_error("Fragment can't be larger than a bundle size");
 
-    uint64_t RequiredBundlePadding = computeBundlePadding(F, F->Offset, FSize);
+    uint64_t RequiredBundlePadding = computeBundlePadding(Assembler, F,
+                                                          F->Offset, FSize);
     if (RequiredBundlePadding > UINT8_MAX)
       report_fatal_error("Padding cannot exceed 255 bytes");
     F->setBundlePadding(static_cast<uint8_t>(RequiredBundlePadding));
     F->Offset += RequiredBundlePadding;
   }
 }
 
 /// \brief Write the contents of a fragment to the given object writer. Expects
 ///        a MCEncodedFragment.
 static void writeFragmentContents(const MCFragment &F, MCObjectWriter *OW) {
   const MCEncodedFragment &EF = cast<MCEncodedFragment>(F);
   OW->WriteBytes(EF.getContents());
 }
 
-/// \brief Write the fragment \p F to the output file.
-static void writeFragment(const MCAssembler &Asm, const MCAsmLayout &Layout,
-                          const MCFragment &F) {
-  MCObjectWriter *OW = &Asm.getWriter();
-
-  // FIXME: Embed in fragments instead?
-  uint64_t FragmentSize = Asm.computeFragmentSize(Layout, F);
-
+void MCAssembler::writeFragmentPadding(const MCFragment &F, uint64_t FSize,
+                                       MCObjectWriter *OW) const {
   // Should NOP padding be written out before this fragment?
   unsigned BundlePadding = F.getBundlePadding();
   if (BundlePadding > 0) {
-    assert(Asm.isBundlingEnabled() &&
+    assert(isBundlingEnabled() &&
            "Writing bundle padding with disabled bundling");
     assert(F.hasInstructions() &&
            "Writing bundle padding for a fragment without instructions");
 
-    unsigned TotalLength = BundlePadding + static_cast<unsigned>(FragmentSize);
-    if (F.alignToBundleEnd() && TotalLength > Asm.getBundleAlignSize()) {
+    unsigned TotalLength = BundlePadding + static_cast<unsigned>(FSize);
+    if (F.alignToBundleEnd() && TotalLength > getBundleAlignSize()) {
       // If the padding itself crosses a bundle boundary, it must be emitted
       // in 2 pieces, since even nop instructions must not cross boundaries.
       //             v--------------v   <- BundleAlignSize
       //        v---------v             <- BundlePadding
       // ----------------------------
       // | Prev |####|####|    F    |
       // ----------------------------
       //        ^-------------------^   <- TotalLength
-      unsigned DistanceToBoundary = TotalLength - Asm.getBundleAlignSize();
-      if (!Asm.getBackend().writeNopData(DistanceToBoundary, OW))
+      unsigned DistanceToBoundary = TotalLength - getBundleAlignSize();
+      if (!getBackend().writeNopData(DistanceToBoundary, OW))
           report_fatal_error("unable to write NOP sequence of " +
                              Twine(DistanceToBoundary) + " bytes");
       BundlePadding -= DistanceToBoundary;
     }
-    if (!Asm.getBackend().writeNopData(BundlePadding, OW))
+    if (!getBackend().writeNopData(BundlePadding, OW))
       report_fatal_error("unable to write NOP sequence of " +
                          Twine(BundlePadding) + " bytes");
   }
+}
+
+/// \brief Write the fragment \p F to the output file.
+static void writeFragment(const MCAssembler &Asm, const MCAsmLayout &Layout,
+                          const MCFragment &F) {
+  MCObjectWriter *OW = &Asm.getWriter();
+
+  // FIXME: Embed in fragments instead?
+  uint64_t FragmentSize = Asm.computeFragmentSize(Layout, F);
+
+  Asm.writeFragmentPadding(F, FragmentSize, OW);
 
   // This variable (and its dummy usage) is to participate in the assert at
   // the end of the function.
   uint64_t Start = OW->getStream().tell();
   (void) Start;
 
   ++stats::EmittedFragments;
 

llvm/trunk/lib/MC/MCELFStreamer.cpp

 // This file assembles .s files and emits ELF .o object files.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/MC/MCELFStreamer.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/MC/MCAsmBackend.h"
+#include "llvm/MC/MCAsmLayout.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCAssembler.h"
 #include "llvm/MC/MCCodeEmitter.h"
 #include "llvm/MC/MCContext.h"
 #include "llvm/MC/MCELF.h"
 #include "llvm/MC/MCELFSymbolFlags.h"
 #include "llvm/MC/MCExpr.h"
 #include "llvm/MC/MCInst.h"

 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace llvm;
 
 MCELFStreamer::~MCELFStreamer() {
 }
 
+void MCELFStreamer::mergeFragment(MCDataFragment *DF,
+                                  MCEncodedFragmentWithFixups *EF) {
+  MCAssembler &Assembler = getAssembler();
+
+  if (Assembler.isBundlingEnabled() && Assembler.getRelaxAll()) {
+    uint64_t FSize = EF->getContents().size();
+
+    if (FSize > Assembler.getBundleAlignSize())
+      report_fatal_error("Fragment can't be larger than a bundle size");
+
+    uint64_t RequiredBundlePadding = computeBundlePadding(
+        Assembler, EF, DF->getContents().size(), FSize);
+
+    if (RequiredBundlePadding > UINT8_MAX)
+      report_fatal_error("Padding cannot exceed 255 bytes");
+
+    if (RequiredBundlePadding > 0) {
+      SmallString<256> Code;
+      raw_svector_ostream VecOS(Code);
+      MCObjectWriter *OW = Assembler.getBackend().createObjectWriter(VecOS);
+
+      EF->setBundlePadding(static_cast<uint8_t>(RequiredBundlePadding));
+
+      Assembler.writeFragmentPadding(*EF, FSize, OW);
+      VecOS.flush();
+      delete OW;
+
+      DF->getContents().append(Code.begin(), Code.end());
+    }
+  }
+
+  flushPendingLabels(DF, DF->getContents().size());
+
+  for (unsigned i = 0, e = EF->getFixups().size(); i != e; ++i) {
+    EF->getFixups()[i].setOffset(EF->getFixups()[i].getOffset() +
+                                 DF->getContents().size());
+    DF->getFixups().push_back(EF->getFixups()[i]);
+  }
+  DF->setHasInstructions(true);
+  DF->getContents().append(EF->getContents().begin(), EF->getContents().end());
+}
+
 void MCELFStreamer::InitSections(bool NoExecStack) {
   // This emulates the same behavior of GNU as. This makes it easier
   // to compare the output as the major sections are in the same order.
   MCContext &Ctx = getContext();
   SwitchSection(Ctx.getObjectFileInfo()->getTextSection());
   EmitCodeAlignment(4);
 
   SwitchSection(Ctx.getObjectFileInfo()->getDataSection());
void MCELFStreamer::EmitInstToData(const MCInst &Inst,
   // - If we're in a bundle-locked group, append the instruction to the current
   //   data fragment because we want all the instructions in a group to get into
   //   the same fragment. Be careful not to do that for the first instruction in
   //   the group, though.
   MCDataFragment *DF;
 
   if (Assembler.isBundlingEnabled()) {
     MCSectionData *SD = getCurrentSectionData();
-    if (SD->isBundleLocked() && !SD->isBundleGroupBeforeFirstInst())
+    if (Assembler.getRelaxAll() && SD->isBundleLocked())
+      // If the -mc-relax-all flag is used and we are bundle-locked, we re-use
+      // the current bundle group.
+      DF = BundleGroups.back();
+    else if (Assembler.getRelaxAll() && !SD->isBundleLocked())
+      // When not in a bundle-locked group and the -mc-relax-all flag is used,
+      // we create a new temporary fragment which will be later merged into
+      // the current fragment.
+      DF = new MCDataFragment();
+    else if (SD->isBundleLocked() && !SD->isBundleGroupBeforeFirstInst())
       // If we are bundle-locked, we re-use the current fragment.
       // The bundle-locking directive ensures this is a new data fragment.
       DF = cast<MCDataFragment>(getCurrentFragment());
     else if (!SD->isBundleLocked() && Fixups.size() == 0) {
       // Optimize memory usage by emitting the instruction to a
       // MCCompactEncodedInstFragment when not in a bundle-locked group and
       // there are no fixups registered.
       MCCompactEncodedInstFragment *CEIF = new MCCompactEncodedInstFragment();
void MCELFStreamer::EmitInstToData(const MCInst &Inst,
 
   // Add the fixups and data.
   for (unsigned i = 0, e = Fixups.size(); i != e; ++i) {
     Fixups[i].setOffset(Fixups[i].getOffset() + DF->getContents().size());
     DF->getFixups().push_back(Fixups[i]);
   }
   DF->setHasInstructions(true);
   DF->getContents().append(Code.begin(), Code.end());
+
+  if (Assembler.isBundlingEnabled() && Assembler.getRelaxAll()) {
+    MCSectionData *SD = getCurrentSectionData();
+    if (!SD->isBundleLocked()) {
+      mergeFragment(getOrCreateDataFragment(), DF);
+      delete DF;
+    }
+  }
 }
 
 void MCELFStreamer::EmitBundleAlignMode(unsigned AlignPow2) {
   assert(AlignPow2 <= 30 && "Invalid bundle alignment");
   MCAssembler &Assembler = getAssembler();
   if (AlignPow2 > 0 && (Assembler.getBundleAlignSize() == 0 ||
                         Assembler.getBundleAlignSize() == 1U << AlignPow2))
     Assembler.setBundleAlignSize(1U << AlignPow2);
   else
     report_fatal_error(".bundle_align_mode cannot be changed once set");
 }
 
 void MCELFStreamer::EmitBundleLock(bool AlignToEnd) {
   MCSectionData *SD = getCurrentSectionData();
 
   // Sanity checks
   //
   if (!getAssembler().isBundlingEnabled())
     report_fatal_error(".bundle_lock forbidden when bundling is disabled");
 
   if (!SD->isBundleLocked())
     SD->setBundleGroupBeforeFirstInst(true);
 
+  if (getAssembler().getRelaxAll() && !SD->isBundleLocked()) {
+    // TODO: drop the lock state and set directly in the fragment
+    MCDataFragment *DF = new MCDataFragment();
+    BundleGroups.push_back(DF);
+  }
+
   SD->setBundleLockState(AlignToEnd ? MCSectionData::BundleLockedAlignToEnd :
                                       MCSectionData::BundleLocked);
 }
 
 void MCELFStreamer::EmitBundleUnlock() {
   MCSectionData *SD = getCurrentSectionData();
 
   // Sanity checks
   if (!getAssembler().isBundlingEnabled())
     report_fatal_error(".bundle_unlock forbidden when bundling is disabled");
   else if (!SD->isBundleLocked())
     report_fatal_error(".bundle_unlock without matching lock");
   else if (SD->isBundleGroupBeforeFirstInst())
     report_fatal_error("Empty bundle-locked group is forbidden");
 
+  // When the -mc-relax-all flag is used, we emit instructions to fragments
+  // stored on a stack. When the bundle unlock is emited, we pop a fragment 
+  // from the stack a merge it to the one below.
+  if (getAssembler().getRelaxAll()) {
+    assert(!BundleGroups.empty() && "There are no bundle groups");
+    MCDataFragment *DF = BundleGroups.back();
+
+    // FIXME: Use BundleGroups to track the lock state instead.
+    SD->setBundleLockState(MCSectionData::NotBundleLocked);
+
+    // FIXME: Use more separate fragments for nested groups. 
+    if (!SD->isBundleLocked()) {
+      mergeFragment(getOrCreateDataFragment(), DF);
+      BundleGroups.pop_back();
+      delete DF;
+    }
+
+    if (SD->getBundleLockState() != MCSectionData::BundleLockedAlignToEnd)
+      getOrCreateDataFragment()->setAlignToBundleEnd(false);
+  } else
     SD->setBundleLockState(MCSectionData::NotBundleLocked);
 }
 
 void MCELFStreamer::Flush() {
   for (std::vector<LocalCommon>::const_iterator i = LocalCommons.begin(),
                                                 e = LocalCommons.end();
        i != e; ++i) {
     MCSymbolData *SD = i->SD;
     uint64_t Size = i->Size;

llvm/trunk/lib/MC/MCObjectStreamer.cpp

 
 MCObjectStreamer::~MCObjectStreamer() {
   delete &Assembler->getBackend();
   delete &Assembler->getEmitter();
   delete &Assembler->getWriter();
   delete Assembler;
 }
 
-void MCObjectStreamer::flushPendingLabels(MCFragment *F) {
+void MCObjectStreamer::flushPendingLabels(MCFragment *F, uint64_t FOffset) {
   if (PendingLabels.size()) {
     if (!F) {
       F = new MCDataFragment();
       CurSectionData->getFragmentList().insert(CurInsertionPoint, F);
       F->setParent(CurSectionData);
     }
     for (MCSymbolData *SD : PendingLabels) {
       SD->setFragment(F);
-      SD->setOffset(0);
+      SD->setOffset(FOffset);
     }
     PendingLabels.clear();
   }
 }
 
 void MCObjectStreamer::reset() {
   if (Assembler)
     Assembler->reset();
MCFragment *MCObjectStreamer::getCurrentFragment() const {
 
   return nullptr;
 }
 
 MCDataFragment *MCObjectStreamer::getOrCreateDataFragment() {
   MCDataFragment *F = dyn_cast_or_null<MCDataFragment>(getCurrentFragment());
   // When bundling is enabled, we don't want to add data to a fragment that
   // already has instructions (see MCELFStreamer::EmitInstToData for details)
-  if (!F || (Assembler->isBundlingEnabled() && F->hasInstructions())) {
+  if (!F || (Assembler->isBundlingEnabled() && !Assembler->getRelaxAll() &&
+             F->hasInstructions())) {
     F = new MCDataFragment();
     insert(F);
   }
   return F;
 }
 
 void MCObjectStreamer::visitUsedSymbol(const MCSymbol &Sym) {
   Assembler->getOrCreateSymbolData(Sym);
void MCObjectStreamer::EmitLabel(MCSymbol *Symbol) {
   MCStreamer::EmitLabel(Symbol);
 
   MCSymbolData &SD = getAssembler().getOrCreateSymbolData(*Symbol);
   assert(!SD.getFragment() && "Unexpected fragment on symbol data!");
 
   // If there is a current fragment, mark the symbol as pointing into it.
   // Otherwise queue the label and set its fragment pointer when we emit the
   // next fragment.
-  if (auto *F = dyn_cast_or_null<MCDataFragment>(getCurrentFragment())) {
+  auto *F = dyn_cast_or_null<MCDataFragment>(getCurrentFragment());
+  if (F && !(getAssembler().isBundlingEnabled() &&
+             getAssembler().getRelaxAll())) {
     SD.setFragment(F);
     SD.setOffset(F->getContents().size());
   } else {
     PendingLabels.push_back(&SD);
   }
 }
 
 void MCObjectStreamer::EmitULEB128Value(const MCExpr *Value) {
void MCObjectStreamer::EmitInstruction(const MCInst &Inst,
   }
 
   // Otherwise emit to a separate fragment.
   EmitInstToFragment(Inst, STI);
 }
 
 void MCObjectStreamer::EmitInstToFragment(const MCInst &Inst,
                                           const MCSubtargetInfo &STI) {
+  if (getAssembler().getRelaxAll() && getAssembler().isBundlingEnabled())
+    llvm_unreachable("All instructions should have already been relaxed");
+
   // Always create a new, separate fragment here, because its size can change
   // during relaxation.
   MCRelaxableFragment *IF = new MCRelaxableFragment(Inst, STI);
   insert(IF);
 
   SmallString<128> Code;
   raw_svector_ostream VecOS(Code);
   getAssembler().getEmitter().EncodeInstruction(Inst, VecOS, IF->getFixups(),

llvm/trunk/test/MC/X86/AlignedBundling/bundle-group-too-large-error.s

 # RUN: not llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu %s -o - 2>&1 | FileCheck %s
+# RUN: not llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu -mc-relax-all %s -o - 2>&1 | FileCheck %s
 
 # CHECK: ERROR: Fragment can't be larger than a bundle size
 
   .text
 foo:
   .bundle_align_mode 4
   pushq   %rbp
 
   .bundle_lock
   pushq   %r14
   callq   bar
   callq   bar
   callq   bar
   callq   bar
   .bundle_unlock
 

llvm/trunk/test/MC/X86/AlignedBundling/different-sections.s

 # RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu %s -o - \
 # RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
+# RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu -mc-relax-all %s -o - \
+# RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
 
 # Test two different executable sections with bundling.
 
   .bundle_align_mode 3
   .section text1, "x"
 # CHECK: section text1
   imull $17, %ebx, %ebp
   imull $17, %ebx, %ebp

llvm/trunk/test/MC/X86/AlignedBundling/labeloffset.s

 # RUN: llvm-mc -triple=i686-linux -filetype=obj %s -o - | \
 # RUN: llvm-objdump -disassemble -no-show-raw-insn -r - | FileCheck %s
 # RUN: llvm-mc -triple=i686-nacl -filetype=obj %s -o - | \
 # RUN: llvm-objdump -disassemble -no-show-raw-insn -r - | FileCheck %s
+# RUN: llvm-mc -triple=i686-nacl -filetype=obj -mc-relax-all %s -o - | \
+# RUN: llvm-objdump -disassemble -no-show-raw-insn -r - | FileCheck %s
 
         .bundle_align_mode 5
         .text
         .globl  main
         .align  32, 0x90
         .type   main,@function
 main:                                   # @main
 # CHECK-LABEL: main:

llvm/trunk/test/MC/X86/AlignedBundling/long-nop-pad.s

 # RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu %s -o - \
 # RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
+# RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu -mc-relax-all %s -o - \
+# RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
 
 # Test that long nops are generated for padding where possible.
 
   .text
 foo:
   .bundle_align_mode 5
 
 # This callq instruction is 5 bytes long

llvm/trunk/test/MC/X86/AlignedBundling/nesting.s

 # RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu %s -o - \
 # RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
+# RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu -mc-relax-all %s -o - \
+# RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
 
 # Will be bundle-aligning to 16 byte boundaries
   .bundle_align_mode 4
   .text
 # CHECK-LABEL: foo
 foo:
 # Test that bundle alignment mode can be set more than once.
   .bundle_align_mode 4

llvm/trunk/test/MC/X86/AlignedBundling/pad-align-to-bundle-end.s

 # RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu %s -o - \
 # RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
+# RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu -mc-relax-all %s -o - \
+# RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
 
 # Test some variations of padding to the end of a bundle.
 
   .text
 foo:
   .bundle_align_mode 4
 
 # Each of these callq instructions is 5 bytes long

llvm/trunk/test/MC/X86/AlignedBundling/pad-bundle-groups.s

 # RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu %s -o - \
 # RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
+# RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu -mc-relax-all %s -o - \
+# RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
 
 # Test some variations of padding for bundle-locked groups.
 
   .text
 foo:
   .bundle_align_mode 4
 
 # Each of these callq instructions is 5 bytes long

llvm/trunk/test/MC/X86/AlignedBundling/relax-at-bundle-end.s

 # RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu %s -o - \
 # RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
+# RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu -mc-relax-all %s -o - \
+# RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
 
 # Test that an instruction near a bundle end gets properly padded
 # after it is relaxed.
 .text
 foo:
         .bundle_align_mode 5
         .rept 29
         push %rax
         .endr
 # CHECK: 1c: push
 # CHECK: 1d: nop
 # CHECK: 20: jne
         jne 0x100
 

llvm/trunk/test/MC/X86/AlignedBundling/relax-in-bundle-group.s

 # RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu %s -o - \
 # RUN:   | llvm-objdump -disassemble - | FileCheck %s
+# RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu -mc-relax-all %s -o - \
+# RUN:   | llvm-objdump -disassemble - | FileCheck %s
 
 # Test that instructions inside bundle-locked groups are relaxed even if their
 # fixup is short enough not to warrant relaxation on its own.
 
   .text
 foo:
   .bundle_align_mode 4
   pushq   %rbp

llvm/trunk/test/MC/X86/AlignedBundling/single-inst-bundling.s

 # RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu %s -o - \
-# RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck %s
+# RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck -check-prefix=CHECK -check-prefix=CHECK-OPT %s
+# RUN: llvm-mc -filetype=obj -triple x86_64-pc-linux-gnu -mc-relax-all %s -o - \
+# RUN:   | llvm-objdump -disassemble -no-show-raw-insn - | FileCheck -check-prefix=CHECK -check-prefix=CHECK-RELAX %s
 
 # Test simple NOP insertion for single instructions.
 
   .text
 foo:
   # Will be bundle-aligning to 16 byte boundaries
   .bundle_align_mode 4
   pushq   %rbp
   pushq   %r14
   pushq   %rbx
 
   movl    %edi, %ebx
   callq   bar
   movl    %eax, %r14d
 
   imull   $17, %ebx, %ebp
 # This imull is 3 bytes long and should have started at 0xe, so two bytes
 # of nop padding are inserted instead and it starts at 0x10
 # CHECK:          nop
 # CHECK-NEXT:     10: imull
 
   movl    %ebx, %edi
   callq   bar
   cmpl    %r14d, %ebp
+# CHECK-RELAX:   nopl
   jle     .L_ELSE
 # Due to the padding that's inserted before the addl, the jump target
 # becomes farther by one byte.
-# CHECK:         jle 5
+# CHECK-OPT:     jle 5
+# CHECK-RELAX:   jle 7
 
   addl    %ebp, %eax
-# CHECK:          nop
-# CHECK-NEXT:     20: addl
+# CHECK-OPT:     nop
+# CHECK-OPT-NEXT:20: addl
+# CHECK-RELAX:   26: addl
 
   jmp     .L_RET
 .L_ELSE:
   imull   %ebx, %eax
 .L_RET:
   ret
 
 # Just sanity checking that data fills don't drive bundling crazy
   .data
   .byte 40
   .byte 98