Provide support for preserving assembly comments

[niravd]

Summary

Preserve assembly comments from input in output assembly and flags to
toggle property. This is on by default for inline assembly and off in
llvm-mc.

Parsed comments are emitted immediately before an EOL which generally
places them on the expected line.

[niravd]

Add llvm-mc test and deal with new LineComment structure

[niravd]

Transform preserved non-assembly comments to assembly-style comments

[majnemer]

What is the rationale behind adding this functionality to MC?

include/llvm/MC/MCParser/MCAsmLexer.h

27–85

Please do not reformat this enum in this differential.

[niravd]

In D20020#452526, @majnemer wrote:

What is the rationale behind adding this functionality to MC?

There are some projects which rely on emitted comments from inline assembly, e.g., Seabios. As a result when compiling with clang they turn off the integrated assembler.

[niravd]

Resolve comments. Change C comment handling to generate better multi-line comments.

[niravd]

Ping.

[rnk]

Hm, sorry, I never hit submit.

In D20020#452579, @niravd wrote:

In D20020#452526, @majnemer wrote:

What is the rationale behind adding this functionality to MC?

There are some projects which rely on emitted comments from inline assembly, e.g., Seabios. As a result when compiling with clang they turn off the integrated assembler.

I think we should support this functionality so that we can give a reasonable workaround to the Linux kernel people for this: https://llvm.org/bugs/show_bug.cgi?id=18891

include/llvm/MC/MCStreamer.h

281

This is only called from MCAsmStreamer. Should we sink it down there and devirtualize it, or do we think there will be clients other than EmitEOL?

include/llvm/Target/TargetOptions.h

191

This should really live in MCTargetOptions next to AsmVerbose, IMO.

lib/MC/MCAsmInfo.cpp

110

Why default to true?

[niravd]

Update in light of rnk's comments

[niravd]

include/llvm/MC/MCStreamer.h

281

I'm inclined to keep it here with the other outputting functions. That said, by that argument it probably should be renamed emitExplicitComments.

include/llvm/Target/TargetOptions.h

191

Make sense. Changed.

191

Agreed

lib/MC/MCAsmInfo.cpp

110

Since the integrated assembler goes from preprocessor assembly to preprocessed assembly with some nominal cleanup, it seems like if we're assuming comments may have semantic value the default should be preservation for people who haven't reasoned about it explicitly.

[rnk]

lgtm

lib/MC/MCAsmInfo.cpp

110

Also, I guess it's basically off by default if you emit an object file. OK.

include/llvm/MC/MCAsmInfo.h

protected:
 
   /// Should we use the integrated assembler?
   /// The integrated assembler should be enabled by default (by the
   /// constructors) when failing to parse a valid piece of assembly (inline
   /// or otherwise) is considered a bug. It may then be overridden after
   /// construction (see LLVMTargetMachine::initAsmInfo()).
   bool UseIntegratedAssembler;
 
+  /// Preserve Comments in assembly
+  bool PreserveAsmComments;
+
   /// Compress DWARF debug sections. Defaults to no compression.
   DebugCompressionType CompressDebugSections;
 
   /// True if the integrated assembler should interpret 'a >> b' constant
   /// expressions as logical rather than arithmetic.
   bool UseLogicalShr;
 
   // If true, emit GOTPCRELX/REX_GOTPCRELX instead of GOTPCREL, on
public:
   /// Return true if assembly (inline or otherwise) should be parsed.
   bool useIntegratedAssembler() const { return UseIntegratedAssembler; }
 
   /// Set whether assembly (inline or otherwise) should be parsed.
   virtual void setUseIntegratedAssembler(bool Value) {
     UseIntegratedAssembler = Value;
   }
 
+  /// Return true if assembly (inline or otherwise) should be parsed.
+  bool preserveAsmComments() const { return PreserveAsmComments; }
+
+  /// Set whether assembly (inline or otherwise) should be parsed.
+  virtual void setPreserveAsmComments(bool Value) {
+    PreserveAsmComments = Value;
+  }
+
   DebugCompressionType compressDebugSections() const {
     return CompressDebugSections;
   }
 
   void setCompressDebugSections(DebugCompressionType CompressDebugSections) {
     this->CompressDebugSections = CompressDebugSections;
   }
 
   bool shouldUseLogicalShr() const { return UseLogicalShr; }
 
   bool canRelaxRelocations() const { return RelaxELFRelocations; }
   void setRelaxELFRelocations(bool V) { RelaxELFRelocations = V; }
 };
 }
 
 #endif

include/llvm/MC/MCParser/AsmLexer.h

 
 /// AsmLexer - Lexer class for assembly files.
 class AsmLexer : public MCAsmLexer {
   const MCAsmInfo &MAI;
 
   const char *CurPtr;
   StringRef CurBuf;
   bool isAtStartOfLine;
+  bool isAtStartOfStatement;
 
   void operator=(const AsmLexer&) = delete;
   AsmLexer(const AsmLexer&) = delete;
 
 protected:
   /// LexToken - Read the next token and return its code.
   AsmToken LexToken() override;
 
 public:
   AsmLexer(const MCAsmInfo &MAI);
   ~AsmLexer() override;
 
   void setBuffer(StringRef Buf, const char *ptr = nullptr);
 
   StringRef LexUntilEndOfStatement() override;
-  StringRef LexUntilEndOfLine();
 
   size_t peekTokens(MutableArrayRef<AsmToken> Buf,
                     bool ShouldSkipSpace = true) override;
 
-  bool isAtStartOfComment(const char *Ptr);
-  bool isAtStatementSeparator(const char *Ptr);
-
   const MCAsmInfo &getMAI() const { return MAI; }
 
 private:
+  bool isAtStartOfComment(const char *Ptr);
+  bool isAtStatementSeparator(const char *Ptr);
   int getNextChar();
   AsmToken ReturnError(const char *Loc, const std::string &Msg);
 
   AsmToken LexIdentifier();
   AsmToken LexSlash();
   AsmToken LexLineComment();
   AsmToken LexDigit();
   AsmToken LexSingleQuote();
   AsmToken LexQuote();
   AsmToken LexFloatLiteral();
   AsmToken LexHexFloatLiteral(bool NoIntDigits);
+
+  StringRef LexUntilEndOfLine();
 };
 
 } // end namespace llvm
 
 #endif

include/llvm/MC/MCParser/MCAsmLexer.h

 #include "llvm/Support/SMLoc.h"
 #include <utility>
 
 namespace llvm {
 
 /// Target independent representation for an assembler token.
 class AsmToken {
 public:
   enum TokenKind {
     // Markers
-    Eof, Error,
+    Eof,
+    Error,
 
     // String values.
     Identifier,
     String,
 
     // Integer values.
     Integer,
     BigNum, // larger than 64 bits
 
     // Real values.
     Real,
 
+    // Comments
+    Comment,
+    HashDirective,
     // No-value.
     EndOfStatement,
     Colon,
     Space,
-    Plus, Minus, Tilde,
-    Slash,    // '/'
+    Plus,
+    Minus,
+    Tilde,
+    Slash,     // '/'
     BackSlash, // '\'
-    LParen, RParen, LBrac, RBrac, LCurly, RCurly,
-    Star, Dot, Comma, Dollar, Equal, EqualEqual,
-
-    Pipe, PipePipe, Caret,
-    Amp, AmpAmp, Exclaim, ExclaimEqual, Percent, Hash,
-    Less, LessEqual, LessLess, LessGreater,
-    Greater, GreaterEqual, GreaterGreater, At
+    LParen,
+    RParen,
+    LBrac,
+    RBrac,
+    LCurly,
+    RCurly,
+    Star,
+    Dot,
+    Comma,
+    Dollar,
+    Equal,
+    EqualEqual,
+
+    Pipe,
+    PipePipe,
+    Caret,
+    Amp,
+    AmpAmp,
+    Exclaim,
+    ExclaimEqual,
+    Percent,
+    Hash,
+    Less,
+    LessEqual,
+    LessLess,
+    LessGreater,
+    Greater,
+    GreaterEqual,
+    GreaterGreater,
+    At
   };
 
 private:
   TokenKind Kind;
 
   /// A reference to the entire token contents; this is always a pointer into
   /// a memory buffer owned by the source manager.
   StringRef Str;
 
public:
 
   /// Consume the next token from the input stream and return it.
   ///
   /// The lexer will continuosly return the end-of-file token once the end of
   /// the main input file has been reached.
   const AsmToken &Lex() {
     assert(!CurTok.empty());
     CurTok.erase(CurTok.begin());
+    // Always place in front as LexToken may generate multiple tokens via UnLex.
     if (CurTok.empty())
-      CurTok.emplace_back(LexToken());
+      CurTok.insert(CurTok.begin(), LexToken());
     return CurTok.front();
   }
 
   void UnLex(AsmToken const &Token) {
     CurTok.insert(CurTok.begin(), Token);
   }
 
   virtual StringRef LexUntilEndOfStatement() = 0;

include/llvm/MC/MCStreamer.h

public:
   /// \brief Return true if this asm streamer supports emitting unformatted text
   /// to the .s file with EmitRawText.
   virtual bool hasRawTextSupport() const { return false; }
 
   /// \brief Is the integrated assembler required for this streamer to function
   /// correctly?
   virtual bool isIntegratedAssemblerRequired() const { return false; }
 
-  /// \brief Add a textual command.
+  /// \brief Add a textual comment.
   ///
   /// Typically for comments that can be emitted to the generated .s
   /// file if applicable as a QoI issue to make the output of the compiler
   /// more readable.  This only affects the MCAsmStreamer, and only when
   /// verbose assembly output is enabled.
   ///
   /// If the comment includes embedded \n's, they will each get the comment
   /// prefix as appropriate.  The added comment should not end with a \n.
   virtual void AddComment(const Twine &T) {}
 
   /// \brief Return a raw_ostream that comments can be written to. Unlike
   /// AddComment, you are required to terminate comments with \n if you use this
   /// method.
   virtual raw_ostream &GetCommentOS();
 
   /// \brief Print T and prefix it with the comment string (normally #) and
   /// optionally a tab. This prints the comment immediately, not at the end of
   /// the current line. It is basically a safe version of EmitRawText: since it
   /// only prints comments, the object streamer ignores it instead of asserting.
   virtual void emitRawComment(const Twine &T, bool TabPrefix = true);
 
+  /// \brief Add explicit comment T. T is required to be a valid
+  /// comment in the output and does not need to be escaped.
+  virtual void addExplicitComment(const Twine &T);
+  /// \brief emit explicit comments
+  virtual void outputExplicitComments();
+
   /// AddBlankLine - Emit a blank line to a .s file to pretty it up.
   virtual void AddBlankLine() {}
 
   /// @}
 
   /// \name Symbol & Section Management
   /// @{
 

include/llvm/Target/TargetOptions.h

public:
 
     /// UseInitArray - Use .init_array instead of .ctors for static
     /// constructors.
     unsigned UseInitArray : 1;
 
     /// Disable the integrated assembler.
     unsigned DisableIntegratedAS : 1;
 
+    /// Disable the integrated assembler.
+    unsigned PreserveAsmComments : 1;
+
     /// Compress DWARF debug sections.
     unsigned CompressDebugSections : 1;
 
     unsigned RelaxELFRelocations : 1;
 
     /// Emit functions into separate sections.
     unsigned FunctionSections : 1;
 

lib/CodeGen/LLVMTargetMachine.cpp

void LLVMTargetMachine::initAsmInfo() {
   // Provide the user with a useful error message about what's wrong.
   assert(TmpAsmInfo && "MCAsmInfo not initialized. "
          "Make sure you include the correct TargetSelect.h"
          "and that InitializeAllTargetMCs() is being invoked!");
 
   if (Options.DisableIntegratedAS)
     TmpAsmInfo->setUseIntegratedAssembler(false);
 
+  TmpAsmInfo->setPreserveAsmComments(Options.PreserveAsmComments);
+
   if (Options.CompressDebugSections)
     TmpAsmInfo->setCompressDebugSections(DebugCompressionType::DCT_ZlibGnu);
 
   TmpAsmInfo->setRelaxELFRelocations(Options.RelaxELFRelocations);
 
   if (Options.ExceptionModel != ExceptionHandling::None)
     TmpAsmInfo->setExceptionsType(Options.ExceptionModel);
 

lib/MC/MCAsmInfo.cpp

MCAsmInfo::MCAsmInfo() {
   // - Windows always enables the integrated assembler by default
   //   - MCAsmInfoCOFF is handling this case, should it be MCAsmInfoMicrosoft?
   // - MachO targets always enables the integrated assembler by default
   //   - MCAsmInfoDarwin is handling this case
   // - Generic_GCC toolchains enable the integrated assembler on a per
   //   architecture basis.
   //   - The target subclasses for AArch64, ARM, and X86 handle these cases
   UseIntegratedAssembler = false;
+  PreserveAsmComments = true;
 
   CompressDebugSections = DebugCompressionType::DCT_None;
 }
 
 MCAsmInfo::~MCAsmInfo() {
 }
 
 bool MCAsmInfo::isSectionAtomizableBySymbols(const MCSection &Section) const {

lib/MC/MCAsmStreamer.cpp

 class MCAsmStreamer final : public MCStreamer {
   std::unique_ptr<formatted_raw_ostream> OSOwner;
   formatted_raw_ostream &OS;
   const MCAsmInfo *MAI;
   std::unique_ptr<MCInstPrinter> InstPrinter;
   std::unique_ptr<MCCodeEmitter> Emitter;
   std::unique_ptr<MCAsmBackend> AsmBackend;
 
+  SmallString<128> ExplicitCommentToEmit;
   SmallString<128> CommentToEmit;
   raw_svector_ostream CommentStream;
 
   unsigned IsVerboseAsm : 1;
   unsigned ShowInst : 1;
   unsigned UseDwarfDirectory : 1;
 
   void EmitRegisterName(int64_t Register);
MCAsmStreamer(MCContext &Context, std::unique_ptr<formatted_raw_ostream> os,
         IsVerboseAsm(isVerboseAsm), ShowInst(showInst),
         UseDwarfDirectory(useDwarfDirectory) {
     assert(InstPrinter);
     if (IsVerboseAsm)
         InstPrinter->setCommentStream(CommentStream);
   }
 
   inline void EmitEOL() {
+    // Dump Explicit Comments here.
+    outputExplicitComments();
     // If we don't have any comments, just emit a \n.
     if (!IsVerboseAsm) {
       OS << '\n';
       return;
     }
     EmitCommentsAndEOL();
   }
 
public:
   raw_ostream &GetCommentOS() override {
     if (!IsVerboseAsm)
       return nulls();  // Discard comments unless in verbose asm mode.
     return CommentStream;
   }
 
   void emitRawComment(const Twine &T, bool TabPrefix = true) override;
 
+  void addExplicitComment(const Twine &T) override;
+  void outputExplicitComments() override;
+
   /// AddBlankLine - Emit a blank line to a .s file to pretty it up.
   void AddBlankLine() override {
     EmitEOL();
   }
 
   /// @name MCStreamer Interface
   /// @{
 

 
 void MCAsmStreamer::emitRawComment(const Twine &T, bool TabPrefix) {
   if (TabPrefix)
     OS << '\t';
   OS << MAI->getCommentString() << T;
   EmitEOL();
 }
 
+void MCAsmStreamer::addExplicitComment(const Twine &T) {
+  StringRef c = T.getSingleStringRef();
+  if (c.equals(StringRef(MAI->getSeparatorString())))
+    return;
+  if (c.startswith(StringRef("//"))) {
+    ExplicitCommentToEmit.append(MAI->getCommentString());
+    // drop //
+    ExplicitCommentToEmit.append(c.slice(2, c.size()).str());
+  } else if (c.startswith(StringRef("/*"))) {
+    size_t p = 2, len = c.size() - 2;
+    // emit each line in comment as separate newline.
+    do {
+      size_t newp = std::min(len, c.find_first_of("\r\n", p));
+      ExplicitCommentToEmit.append(MAI->getCommentString());
+      ExplicitCommentToEmit.append(c.slice(p, newp).str());
+      p = newp + 1;
+    } while (p < len);
+  } else if (c.startswith(StringRef(MAI->getCommentString()))) {
+    ExplicitCommentToEmit.append(c.str());
+  } else if (c.front() == '#') {
+    // # are comments for ## commentString. Output extra #.
+    ExplicitCommentToEmit.append("#");
+    ExplicitCommentToEmit.append(c.str());
+  } else
+    assert(false && "Unexpected Assembly Comment");
+
+  // full line comments immediately output
+  if (c.back() == '\n')
+    outputExplicitComments();
+}
+
+void MCAsmStreamer::outputExplicitComments() {
+  StringRef Comments = ExplicitCommentToEmit;
+  if (!Comments.empty())
+    OS << '\t' << Comments;
+  ExplicitCommentToEmit.clear();
+}
+
 void MCAsmStreamer::ChangeSection(MCSection *Section,
                                   const MCExpr *Subsection) {
   assert(Section && "Cannot switch to a null section!");
   Section->PrintSwitchToSection(*MAI, OS, Subsection);
 }
 
 void MCAsmStreamer::EmitLabel(MCSymbol *Symbol) {
   assert(Symbol->isUndefined() && "Cannot define a symbol twice!");

 void MCAsmStreamer::EmitSymbolDesc(MCSymbol *Symbol, unsigned DescValue) {
   OS << ".desc" << ' ';
   Symbol->print(OS, MAI);
   OS << ',' << DescValue;
   EmitEOL();
 }
 
 void MCAsmStreamer::EmitSyntaxDirective() {
-  if (MAI->getAssemblerDialect() == 1)
-    OS << "\t.intel_syntax noprefix\n";
+  if (MAI->getAssemblerDialect() == 1) {
+    OS << "\t.intel_syntax noprefix";
+    EmitEOL();
+  }
   // FIXME: Currently emit unprefix'ed registers.
   // The intel_syntax directive has one optional argument 
   // with may have a value of prefix or noprefix.
 }
 
 void MCAsmStreamer::BeginCOFFSymbolDef(const MCSymbol *Symbol) {
   OS << "\t.def\t ";
   Symbol->print(OS, MAI);

lib/MC/MCParser/AsmLexer.cpp

 #include <cerrno>
 #include <cstdio>
 #include <cstdlib>
 using namespace llvm;
 
 AsmLexer::AsmLexer(const MCAsmInfo &MAI) : MAI(MAI) {
   CurPtr = nullptr;
   isAtStartOfLine = true;
+  isAtStartOfStatement = true;
   AllowAtInIdentifier = !StringRef(MAI.getCommentString()).startswith("@");
 }
 
 AsmLexer::~AsmLexer() {
 }
 
 void AsmLexer::setBuffer(StringRef Buf, const char *ptr) {
   CurBuf = Buf;
 
   if (ptr)
     CurPtr = ptr;
   else
     CurPtr = CurBuf.begin();
 
   TokStart = nullptr;
 }
 
 /// ReturnError - Set the error to the specified string at the specified
 /// location.  This is defined to always return AsmToken::Error.
 AsmToken AsmLexer::ReturnError(const char *Loc, const std::string &Msg) {
   SetError(SMLoc::getFromPointer(Loc), Msg);
 
-  return AsmToken(AsmToken::Error, StringRef(Loc, 0));
+  return AsmToken(AsmToken::Error, StringRef(Loc, CurPtr - Loc));
 }
 
 int AsmLexer::getNextChar() {
-  char CurChar = *CurPtr++;
-  switch (CurChar) {
-  default:
-    return (unsigned char)CurChar;
-  case 0:
-    // A nul character in the stream is either the end of the current buffer or
-    // a random nul in the file.  Disambiguate that here.
-    if (CurPtr - 1 != CurBuf.end())
-      return 0;  // Just whitespace.
-
-    // Otherwise, return end of file.
-    --CurPtr;  // Another call to lex will return EOF again.
+  if (CurPtr == CurBuf.end())
     return EOF;
-  }
+  return (unsigned char)*CurPtr++;
 }
 
 /// LexFloatLiteral: [0-9]*[.][0-9]*([eE][+-]?[0-9]*)?
 ///
 /// The leading integral digit sequence and dot should have already been
 /// consumed, some or all of the fractional digit sequence *can* have been
 /// consumed.
 AsmToken AsmLexer::LexFloatLiteral() {
AsmToken AsmLexer::LexIdentifier() {
   return AsmToken(AsmToken::Identifier, StringRef(TokStart, CurPtr - TokStart));
 }
 
 /// LexSlash: Slash: /
 ///           C-Style Comment: /* ... */
 AsmToken AsmLexer::LexSlash() {
   switch (*CurPtr) {
   case '*':
+    isAtStartOfStatement = false;
     break; // C style comment.
   case '/':
     ++CurPtr;
     return LexLineComment();
   default:
-    return AsmToken(AsmToken::Slash, StringRef(CurPtr - 1, 1));
+    isAtStartOfStatement = false;
+    return AsmToken(AsmToken::Slash, StringRef(TokStart, 1));
   }
 
   // C Style comment.
   ++CurPtr;  // skip the star.
-  while (1) {
-    int CurChar = getNextChar();
-    switch (CurChar) {
-    case EOF:
-      return ReturnError(TokStart, "unterminated comment");
+  while (CurPtr != CurBuf.end()) {
+    switch (*CurPtr++) {
     case '*':
       // End of the comment?
-      if (CurPtr[0] != '/') break;
-
+      if (*CurPtr != '/')
+        break;
       ++CurPtr;   // End the */.
-      return LexToken();
+      return AsmToken(AsmToken::Comment,
+                      StringRef(TokStart, CurPtr - TokStart));
     }
   }
+  return ReturnError(TokStart, "unterminated comment");
 }
 
 /// LexLineComment: Comment: #[^\n]*
 ///                        : //[^\n]*
 AsmToken AsmLexer::LexLineComment() {
-  // FIXME: This is broken if we happen to a comment at the end of a file, which
-  // was .included, and which doesn't end with a newline.
+  // Mark This as an end of statement with a body of the
+  // comment. While it would be nicer to leave this two tokens,
+  // backwards compatability with TargetParsers makes keeping this in this form
+  // better.
   int CurChar = getNextChar();
   while (CurChar != '\n' && CurChar != '\r' && CurChar != EOF)
     CurChar = getNextChar();
 
-  if (CurChar == EOF)
-    return AsmToken(AsmToken::Eof, StringRef(TokStart, 0));
-  return AsmToken(AsmToken::EndOfStatement, StringRef(TokStart, 0));
+  // If this is a whole line comment. leave newline
+  bool keepnewline = isAtStartOfStatement;
+  isAtStartOfLine = true;
+  isAtStartOfStatement = true;
+
+  if (keepnewline)
+    return AsmToken(AsmToken::EndOfStatement,
+                    StringRef(TokStart, CurPtr - TokStart));
+  return AsmToken(AsmToken::EndOfStatement,
+                  StringRef(TokStart, CurPtr - 1 - TokStart));
 }
 
 static void SkipIgnoredIntegerSuffix(const char *&CurPtr) {
   // Skip ULL, UL, U, L and LL suffices.
   if (CurPtr[0] == 'U')
     ++CurPtr;
   if (CurPtr[0] == 'L')
     ++CurPtr;
AsmToken AsmLexer::LexQuote() {
   return AsmToken(AsmToken::String, StringRef(TokStart, CurPtr - TokStart));
 }
 
 StringRef AsmLexer::LexUntilEndOfStatement() {
   TokStart = CurPtr;
 
   while (!isAtStartOfComment(CurPtr) &&     // Start of line comment.
          !isAtStatementSeparator(CurPtr) && // End of statement marker.
-         *CurPtr != '\n' && *CurPtr != '\r' &&
-         (*CurPtr != 0 || CurPtr != CurBuf.end())) {
+         *CurPtr != '\n' && *CurPtr != '\r' && CurPtr != CurBuf.end()) {
     ++CurPtr;
   }
   return StringRef(TokStart, CurPtr-TokStart);
 }
 
 StringRef AsmLexer::LexUntilEndOfLine() {
   TokStart = CurPtr;
 
-  while (*CurPtr != '\n' && *CurPtr != '\r' &&
-         (*CurPtr != 0 || CurPtr != CurBuf.end())) {
+  while (*CurPtr != '\n' && *CurPtr != '\r' && CurPtr != CurBuf.end()) {
     ++CurPtr;
   }
   return StringRef(TokStart, CurPtr-TokStart);
 }
 
 size_t AsmLexer::peekTokens(MutableArrayRef<AsmToken> Buf,
                             bool ShouldSkipSpace) {
   const char *SavedTokStart = TokStart;
   const char *SavedCurPtr = CurPtr;
   bool SavedAtStartOfLine = isAtStartOfLine;
+  bool SavedAtStartOfStatement = isAtStartOfStatement;
   bool SavedSkipSpace = SkipSpace;
 
   std::string SavedErr = getErr();
   SMLoc SavedErrLoc = getErrLoc();
 
   SkipSpace = ShouldSkipSpace;
 
   size_t ReadCount;
   for (ReadCount = 0; ReadCount < Buf.size(); ++ReadCount) {
     AsmToken Token = LexToken();
 
     Buf[ReadCount] = Token;
 
     if (Token.is(AsmToken::Eof))
       break;
   }
 
   SetError(SavedErrLoc, SavedErr);
 
   SkipSpace = SavedSkipSpace;
   isAtStartOfLine = SavedAtStartOfLine;
+  isAtStartOfStatement = SavedAtStartOfStatement;
   CurPtr = SavedCurPtr;
   TokStart = SavedTokStart;
 
   return ReadCount;
 }
 
 bool AsmLexer::isAtStartOfComment(const char *Ptr) {
   const char *CommentString = MAI.getCommentString();
return strncmp(Ptr, MAI.getSeparatorString(),
                  strlen(MAI.getSeparatorString())) == 0;
 }
 
 AsmToken AsmLexer::LexToken() {
   TokStart = CurPtr;
   // This always consumes at least one character.
   int CurChar = getNextChar();
 
-  if (isAtStartOfComment(TokStart)) {
-    // If this comment starts with a '#', then return the Hash token and let
-    // the assembler parser see if it can be parsed as a cpp line filename
-    // comment. We do this only if we are at the start of a line.
-    if (CurChar == '#' && isAtStartOfLine)
-      return AsmToken(AsmToken::Hash, StringRef(TokStart, 1));
-    isAtStartOfLine = true;
+  if (CurChar == '#' && isAtStartOfStatement) {
+    // If this starts with a '#', this may be a cpp
+    // hash directive and otherwise a line comment.
+    AsmToken TokenBuf[2];
+    MutableArrayRef<AsmToken> Buf(TokenBuf, 2);
+    size_t num = peekTokens(Buf, true);
+    // There cannot be a space preceeding this
+    if (isAtStartOfLine && num == 2 && TokenBuf[0].is(AsmToken::Integer) &&
+        TokenBuf[1].is(AsmToken::String)) {
+      CurPtr = TokStart; // reset curPtr;
+      StringRef s = LexUntilEndOfLine();
+      UnLex(TokenBuf[1]);
+      UnLex(TokenBuf[0]);
+      return AsmToken(AsmToken::HashDirective, s);
+    }
     return LexLineComment();
   }
+
+  if (isAtStartOfComment(TokStart))
+    return LexLineComment();
+
   if (isAtStatementSeparator(TokStart)) {
     CurPtr += strlen(MAI.getSeparatorString()) - 1;
+    isAtStartOfLine = true;
+    isAtStartOfStatement = true;
     return AsmToken(AsmToken::EndOfStatement,
                     StringRef(TokStart, strlen(MAI.getSeparatorString())));
   }
 
   // If we're missing a newline at EOF, make sure we still get an
   // EndOfStatement token before the Eof token.
-  if (CurChar == EOF && !isAtStartOfLine) {
+  if (CurChar == EOF && !isAtStartOfStatement) {
     isAtStartOfLine = true;
+    isAtStartOfStatement = true;
     return AsmToken(AsmToken::EndOfStatement, StringRef(TokStart, 1));
   }
-
   isAtStartOfLine = false;
+  bool OldIsAtStartOfStatement = isAtStartOfStatement;
+  isAtStartOfStatement = false;
   switch (CurChar) {
   default:
     // Handle identifier: [a-zA-Z_.][a-zA-Z0-9_$.@]*
     if (isalpha(CurChar) || CurChar == '_' || CurChar == '.')
       return LexIdentifier();
 
     // Unknown character, emit an error.
     return ReturnError(TokStart, "invalid character in input");
-  case EOF: return AsmToken(AsmToken::Eof, StringRef(TokStart, 0));
+  case EOF:
+    isAtStartOfLine = true;
+    isAtStartOfStatement = true;
+    return AsmToken(AsmToken::Eof, StringRef(TokStart, 0));
   case 0:
   case ' ':
   case '\t':
-    if (SkipSpace) {
-      // Ignore whitespace.
-      return LexToken();
-    } else {
-      int len = 1;
-      while (*CurPtr==' ' || *CurPtr=='\t') {
-        CurPtr++;
-        len++;
-      }
-      return AsmToken(AsmToken::Space, StringRef(TokStart, len));
-    }
-  case '\n': // FALL THROUGH.
+    isAtStartOfStatement = OldIsAtStartOfStatement;
+    while (*CurPtr == ' ' || *CurPtr == '\t')
+    if (SkipSpace)
+      return LexToken(); // Ignore whitespace.
+    else
+      return AsmToken(AsmToken::Space, StringRef(TokStart, CurPtr - TokStart));
+  case '\n':
   case '\r':
     isAtStartOfLine = true;
+    isAtStartOfStatement = true;
     return AsmToken(AsmToken::EndOfStatement, StringRef(TokStart, 1));
   case ':': return AsmToken(AsmToken::Colon, StringRef(TokStart, 1));
   case '+': return AsmToken(AsmToken::Plus, StringRef(TokStart, 1));
   case '-': return AsmToken(AsmToken::Minus, StringRef(TokStart, 1));
   case '~': return AsmToken(AsmToken::Tilde, StringRef(TokStart, 1));
   case '(': return AsmToken(AsmToken::LParen, StringRef(TokStart, 1));
   case ')': return AsmToken(AsmToken::RParen, StringRef(TokStart, 1));
   case '[': return AsmToken(AsmToken::LBrac, StringRef(TokStart, 1));
case '&':
     return AsmToken(AsmToken::Amp, StringRef(TokStart, 1));
   case '!':
     if (*CurPtr == '=') {
       ++CurPtr;
       return AsmToken(AsmToken::ExclaimEqual, StringRef(TokStart, 2));
     }
     return AsmToken(AsmToken::Exclaim, StringRef(TokStart, 1));
   case '%': return AsmToken(AsmToken::Percent, StringRef(TokStart, 1));
-  case '/': return LexSlash();
+  case '/':
+    isAtStartOfStatement = OldIsAtStartOfStatement;
+    return LexSlash();
   case '#': return AsmToken(AsmToken::Hash, StringRef(TokStart, 1));
   case '\'': return LexSingleQuote();
   case '"': return LexQuote();
   case '0': case '1': case '2': case '3': case '4':
   case '5': case '6': case '7': case '8': case '9':
     return LexDigit();
   case '<':
     switch (*CurPtr) {

lib/MC/MCParser/AsmParser.cpp

public:
   void checkForValidSection() override;
   /// }
 
 private:
 
   bool parseStatement(ParseStatementInfo &Info,
                       MCAsmParserSemaCallback *SI);
   bool parseCurlyBlockScope(SmallVectorImpl<AsmRewrite>& AsmStrRewrites);
-  void eatToEndOfLine();
   bool parseCppHashLineFilenameComment(SMLoc L);
 
   void checkForBadMacro(SMLoc DirectiveLoc, StringRef Name, StringRef Body,
                         ArrayRef<MCAsmMacroParameter> Parameters);
   bool expandMacro(raw_svector_ostream &OS, StringRef Body,
                    ArrayRef<MCAsmMacroParameter> Parameters,
                    ArrayRef<MCAsmMacroArgument> A, bool EnableAtPseudoVariable,
                    SMLoc L);

 
 void AsmParser::jumpToLoc(SMLoc Loc, unsigned InBuffer) {
   CurBuffer = InBuffer ? InBuffer : SrcMgr.FindBufferContainingLoc(Loc);
   Lexer.setBuffer(SrcMgr.getMemoryBuffer(CurBuffer)->getBuffer(),
                   Loc.getPointer());
 }
 
 const AsmToken &AsmParser::Lex() {
+  if (Lexer.getTok().is(AsmToken::Error))
+    Error(Lexer.getErrLoc(), Lexer.getErr());
+
+  // if it's a end of statement with a comment in it
+  if (getTok().is(AsmToken::EndOfStatement)) {
+    // if this is a line comment output it.
+    if (getTok().getString().front() != '\n' &&
+        getTok().getString().front() != '\r' && MAI.preserveAsmComments())
+      Out.addExplicitComment(Twine(getTok().getString()));
+  }
+
   const AsmToken *tok = &Lexer.Lex();
+  // Parse comments here to be deferred until end of next statement.
+  while (tok->is(AsmToken::Comment)) {
+    if (MAI.preserveAsmComments())
+      Out.addExplicitComment(Twine(tok->getString()));
+    tok = &Lexer.Lex();
+  }
 
   if (tok->is(AsmToken::Eof)) {
     // If this is the end of an included file, pop the parent file off the
     // include stack.
     SMLoc ParentIncludeLoc = SrcMgr.getParentIncludeLoc(CurBuffer);
     if (ParentIncludeLoc != SMLoc()) {
       jumpToLoc(ParentIncludeLoc);
-      tok = &Lexer.Lex();
+      return Lex();
     }
   }
 
-  if (tok->is(AsmToken::Error))
-    Error(Lexer.getErrLoc(), Lexer.getErr());
 
   return *tok;
 }
 
 bool AsmParser::Run(bool NoInitialTextSection, bool NoFinalize) {
   // Create the initial section, if requested.
   if (!NoInitialTextSection)
     Out.InitSections(false);
bool AsmParser::Run(bool NoInitialTextSection, bool NoFinalize) {
   }
 
   // While we have input, parse each statement.
   while (Lexer.isNot(AsmToken::Eof)) {
     ParseStatementInfo Info;
     if (!parseStatement(Info, nullptr))
       continue;
 
+    // If we've failed, but on a Error Token, but did not consume it in
+    // favor of a better message, emit it now.
+    if (Lexer.getTok().is(AsmToken::Error)) {
+      Lex();
+    }
+
     // We had an error, validate that one was emitted and recover by skipping to
     // the next line.
     assert(HadError && "Parse statement returned an error, but none emitted!");
     eatToEndOfStatement();
   }
 
   if (TheCondState.TheCond != StartingCondState.TheCond ||
       TheCondState.Ignore != StartingCondState.Ignore)
if (MAI.hasSubsectionsViaSymbols()) {
         // Variable symbols may not be marked as defined, so check those
         // explicitly. If we know it's a variable, we have a definition for
         // the purposes of this check.
         if (Sym->isTemporary() && !Sym->isVariable() && !Sym->isDefined())
           // FIXME: We would really like to refer back to where the symbol was
           // first referenced for a source location. We need to add something
           // to track that. Currently, we just point to the end of the file.
           HadError |=
-              Error(getLexer().getLoc(), "assembler local symbol '" +
-                                             Sym->getName() + "' not defined");
+              Error(getTok().getLoc(), "assembler local symbol '" +
       }
     }
 
     // Temporary symbols like the ones for directional jumps don't go in the
     // symbol table. They also need to be diagnosed in all (final) cases.
     for (std::tuple<SMLoc, CppHashInfoTy, MCSymbol *> &LocSym : DirLabels) {
       if (std::get<2>(LocSym)->isUndefined()) {
         // Reset the state of any "# line file" directives we've seen to the
if (!ParsingInlineAsm && !getStreamer().getCurrentSection().first) {
     TokError("expected section directive before assembly directive");
     Out.InitSections(false);
   }
 }
 
 /// \brief Throw away the rest of the line for testing purposes.
 void AsmParser::eatToEndOfStatement() {
   while (Lexer.isNot(AsmToken::EndOfStatement) && Lexer.isNot(AsmToken::Eof))
-    Lex();
+    Lexer.Lex();
 
   // Eat EOL.
   if (Lexer.is(AsmToken::EndOfStatement))
-    Lex();
+    Lexer.Lex();
 }
 
 StringRef AsmParser::parseStringToEndOfStatement() {
   const char *Start = getTok().getLoc().getPointer();
 
   while (Lexer.isNot(AsmToken::EndOfStatement) && Lexer.isNot(AsmToken::Eof))
-    Lex();
+    Lexer.Lex();
 
   const char *End = getTok().getLoc().getPointer();
   return StringRef(Start, End - Start);
 }
 
 StringRef AsmParser::parseStringToComma() {
   const char *Start = getTok().getLoc().getPointer();
 
   while (Lexer.isNot(AsmToken::EndOfStatement) &&
          Lexer.isNot(AsmToken::Comma) && Lexer.isNot(AsmToken::Eof))
-    Lex();
+    Lexer.Lex();
 
   const char *End = getTok().getLoc().getPointer();
   return StringRef(Start, End - Start);
 }
 
 /// \brief Parse a paren expression and return it.
 /// NOTE: This assumes the leading '(' has already been consumed.
 ///
if (parseIdentifier(Identifier)) {
         return Error(FirstTokenLoc, "invalid token in expression");
       }
     }
     // Parse symbol variant
     std::pair<StringRef, StringRef> Split;
     if (!MAI.useParensForSymbolVariant()) {
       if (FirstTokenKind == AsmToken::String) {
         if (Lexer.is(AsmToken::At)) {
-          Lexer.Lex(); // eat @
+          Lex(); // eat @
           SMLoc AtLoc = getLexer().getLoc();
           StringRef VName;
           if (parseIdentifier(VName))
             return Error(AtLoc, "expected symbol variant after '@'");
 
           Split = std::make_pair(Identifier, VName);
         }
       } else {
         Split = Identifier.split('@');
       }
     } else if (Lexer.is(AsmToken::LParen)) {
-      Lexer.Lex(); // eat (
+      Lex(); // eat '('.
       StringRef VName;
       parseIdentifier(VName);
       if (Lexer.isNot(AsmToken::RParen)) {
           return Error(Lexer.getTok().getLoc(),
                        "unexpected token in variant, expected ')'");
       }
-      Lexer.Lex(); // eat )
+      Lex(); // eat ')'.
       Split = std::make_pair(Identifier, VName);
     }
 
     EndLoc = SMLoc::getFromPointer(Identifier.end());
 
     // This is a symbol reference.
     StringRef SymbolName = Identifier;
     MCSymbolRefExpr::VariantKind Variant = MCSymbolRefExpr::VK_None;

 }
 
 /// ParseStatement:
 ///   ::= EndOfStatement
 ///   ::= Label* Directive ...Operands... EndOfStatement
 ///   ::= Label* Identifier OperandList* EndOfStatement
 bool AsmParser::parseStatement(ParseStatementInfo &Info,
                                MCAsmParserSemaCallback *SI) {
+  // Eat initial spaces and comments
+  while (Lexer.is(AsmToken::Space))
+    Lex();
   if (Lexer.is(AsmToken::EndOfStatement)) {
+    // if this is a line comment we can drop it safely
+    if (getTok().getString().front() == '\r' ||
+        getTok().getString().front() == '\n')
       Out.AddBlankLine();
     Lex();
     return false;
   }
-
-  // Statements always start with an identifier or are a full line comment.
+  // Statements always start with an identifier.
   AsmToken ID = getTok();
   SMLoc IDLoc = ID.getLoc();
   StringRef IDVal;
   int64_t LocalLabelVal = -1;
-  // A full line comment is a '#' as the first token.
-  if (Lexer.is(AsmToken::Hash))
+  if (Lexer.is(AsmToken::HashDirective))
     return parseCppHashLineFilenameComment(IDLoc);
-
   // Allow an integer followed by a ':' as a directional local label.
   if (Lexer.is(AsmToken::Integer)) {
     LocalLabelVal = getTok().getIntVal();
     if (LocalLabelVal < 0) {
       if (!TheCondState.Ignore)
         return TokError("unexpected token at start of statement");
       IDVal = "";
     } else {
if (IDVal[0] == '.' && IDVal != ".") {
     case DK_ABORT:
       return parseDirectiveAbort();
     case DK_INCLUDE:
       return parseDirectiveInclude();
     case DK_INCBIN:
       return parseDirectiveIncbin();
     case DK_CODE16:
     case DK_CODE16GCC:
-      return TokError(Twine(IDVal) + " not supported yet");
+      return TokError(Twine(IDVal) +
+                      " not currently supported for this target");
     case DK_REPT:
       return parseDirectiveRept(IDLoc, IDVal);
     case DK_IRP:
       return parseDirectiveIrp(IDLoc);
     case DK_IRPC:
       return parseDirectiveIrpc(IDLoc);
     case DK_ENDR:
       return parseDirectiveEndr(IDLoc);
if (Lexer.is(AsmToken::EndOfStatement))
     Lex(); // Eat EndOfStatement following the brace
 
   // Erase the block start/end brace from the output asm string
   AsmStrRewrites.emplace_back(AOK_Skip, StartLoc, Lexer.getLoc().getPointer() -
                                                   StartLoc.getPointer());
   return true;
 }
 
-/// eatToEndOfLine uses the Lexer to eat the characters to the end of the line
-/// since they may not be able to be tokenized to get to the end of line token.
-void AsmParser::eatToEndOfLine() {
-  if (!Lexer.is(AsmToken::EndOfStatement))
-    Lexer.LexUntilEndOfLine();
-  // Eat EOL.
-  Lex();
-}
-
 /// parseCppHashLineFilenameComment as this:
 ///   ::= # number "filename"
-/// or just as a full line comment if it doesn't have a number and a string.
 bool AsmParser::parseCppHashLineFilenameComment(SMLoc L) {
   Lex(); // Eat the hash token.
-
-  if (getLexer().isNot(AsmToken::Integer)) {
-    // Consume the line since in cases it is not a well-formed line directive,
-    // as if were simply a full line comment.
-    eatToEndOfLine();
-    return false;
-  }
-
+  // Lexer only ever emits HashDirective if it fully formed if it's
+  // done the checking already so this is an internal error.
+  assert(getTok().is(AsmToken::Integer) &&
+         "Lexing Cpp line comment: Expected Integer");
   int64_t LineNumber = getTok().getIntVal();
   Lex();
-
-  if (getLexer().isNot(AsmToken::String)) {
-    eatToEndOfLine();
-    return false;
-  }
-
+  assert(getTok().is(AsmToken::String) &&
+         "Lexing Cpp line comment: Expected String");
   StringRef Filename = getTok().getString();
+  Lex();
   // Get rid of the enclosing quotes.
   Filename = Filename.substr(1, Filename.size() - 2);
 
   // Save the SMLoc, Filename and LineNumber for later use by diagnostics.
   CppHashInfo.Loc = L;
   CppHashInfo.Filename = Filename;
   CppHashInfo.LineNumber = LineNumber;
   CppHashInfo.Buf = CurBuffer;
-
-  // Ignore any trailing characters, they're just comment.
-  eatToEndOfLine();
   return false;
 }
 
 /// \brief will use the last parsed cpp hash line filename comment
 /// for the Filename and LineNo if any in the diagnostic.
 void AsmParser::DiagHandler(const SMDiagnostic &Diag, void *Context) {
   const AsmParser *Parser = static_cast<const AsmParser *>(Context);
   raw_ostream &OS = errs();
for (;;) {
 
     if (ParenLevel == 0) {
 
       if (Lexer.is(AsmToken::Comma))
         break;
 
       if (Lexer.is(AsmToken::Space)) {
         SpaceEaten = true;
-        Lex(); // Eat spaces
+        Lexer.Lex(); // Eat spaces
       }
 
       // Spaces can delimit parameters, but could also be part an expression.
       // If the token after a space is an operator, add the token and the next
       // one into this argument
       if (!IsDarwin) {
         if (isOperator(Lexer.getKind())) {
           MA.push_back(getTok());
-          Lex();
+          Lexer.Lex();
 
           // Whitespace after an operator can be ignored.
           if (Lexer.is(AsmToken::Space))
-            Lex();
+            Lexer.Lex();
 
           continue;
         }
       }
       if (SpaceEaten)
         break;
     }
 
     // handleMacroEntry relies on not advancing the lexer here
     // to be able to fill in the remaining default parameter values
     if (Lexer.is(AsmToken::EndOfStatement))
       break;
 
     // Adjust the current parentheses level.
     if (Lexer.is(AsmToken::LParen))
       ++ParenLevel;
     else if (Lexer.is(AsmToken::RParen) && ParenLevel)
       --ParenLevel;
 
     // Append the token to the current argument list.
     MA.push_back(getTok());
-    Lex();
+    Lexer.Lex();
   }
 
   if (ParenLevel != 0)
     return TokError("unbalanced parentheses in macro argument");
   return false;
 }
 
 // Parse the macro instantiation arguments.
for (unsigned Parameter = 0; !NParameters || Parameter < NParameters;
     unsigned PI = Parameter;
     if (!FA.Name.empty()) {
       unsigned FAI = 0;
       for (FAI = 0; FAI < NParameters; ++FAI)
         if (M->Parameters[FAI].Name == FA.Name)
           break;
 
       if (FAI >= NParameters) {
         assert(M && "expected macro to be defined");
         Error(IDLoc,
               "parameter named '" + FA.Name + "' does not exist for macro '" +
               M->Name + "'");
         return true;
       }
       PI = FAI;
     }
 
bool AsmParser::parseIdentifier(StringRef &Res) {
   // allow things like '.globl $foo' and '.def @feat.00', which would normally be
   // separate tokens. At this level, we have already lexed so we cannot (currently)
   // handle this as a context dependent token, instead we detect adjacent tokens
   // and return the combined identifier.
   if (Lexer.is(AsmToken::Dollar) || Lexer.is(AsmToken::At)) {
     SMLoc PrefixLoc = getLexer().getLoc();
 
     // Consume the prefix character, and check for a following identifier.
-    Lex();
+    Lexer.Lex(); // Lexer's Lex guarantees consecutive token.
     if (Lexer.isNot(AsmToken::Identifier))
       return true;
 
     // We have a '$' or '@' followed by an identifier, make sure they are adjacent.
     if (PrefixLoc.getPointer() + 1 != getTok().getLoc().getPointer())
       return true;
 
     // Construct the joined identifier and consume the token.
     Res =
         StringRef(PrefixLoc.getPointer(), getTok().getIdentifier().size() + 1);
-    Lex();
+    Lexer.Lex(); // Lexer's Lex guarantees consecutive token
     return false;
   }
 
   if (Lexer.isNot(AsmToken::Identifier) && Lexer.isNot(AsmToken::String))
     return true;
 
   Res = getTok().getIdentifier();
 
bool AsmParser::parseDirectiveReloc(SMLoc DirectiveLoc) {
   if (!Offset->evaluateAsAbsolute(OffsetValue))
     return Error(OffsetLoc, "expression is not a constant value");
 
   if (OffsetValue < 0)
     return Error(OffsetLoc, "expression is negative");
 
   if (Lexer.isNot(AsmToken::Comma))
     return TokError("expected comma");
-  Lexer.Lex();
+  Lex();
 
   if (Lexer.isNot(AsmToken::Identifier))
     return TokError("expected relocation name");
   SMLoc NameLoc = Lexer.getTok().getLoc();
   StringRef Name = Lexer.getTok().getIdentifier();
-  Lexer.Lex();
+  Lex();
 
   if (Lexer.is(AsmToken::Comma)) {
-    Lexer.Lex();
+    Lex();
     SMLoc ExprLoc = Lexer.getLoc();
     if (parseExpression(Expr))
       return true;
 
     MCValue Value;
     if (!Expr->evaluateAsRelocatable(Value, nullptr, nullptr))
       return Error(ExprLoc, "expression must be relocatable");
   }
bool AsmParser::parseDirectiveRealValue(const fltSemantics &Semantics) {
   if (getLexer().isNot(AsmToken::EndOfStatement)) {
     checkForValidSection();
 
     for (;;) {
       // We don't truly support arithmetic on floating point expressions, so we
       // have to manually parse unary prefixes.
       bool IsNeg = false;
       if (getLexer().is(AsmToken::Minus)) {
-        Lex();
+        Lexer.Lex();
         IsNeg = true;
       } else if (getLexer().is(AsmToken::Plus))
-        Lex();
+        Lexer.Lex();
 
-      if (getLexer().isNot(AsmToken::Integer) &&
-          getLexer().isNot(AsmToken::Real) &&
-          getLexer().isNot(AsmToken::Identifier))
+      if (Lexer.is(AsmToken::Error))
+        return TokError(Lexer.getErr());
+      if (Lexer.isNot(AsmToken::Integer) && Lexer.isNot(AsmToken::Real) &&
+          Lexer.isNot(AsmToken::Identifier))
         return TokError("unexpected token in directive");
 
       // Convert to an APFloat.
       APFloat Value(Semantics);
       StringRef IDVal = getTok().getString();
       if (getLexer().is(AsmToken::Identifier)) {
         if (!IDVal.compare_lower("infinity") || !IDVal.compare_lower("inf"))
           Value = APFloat::getInf(Semantics);
for (;;) {
       // Consume the numeric token.
       Lex();
 
       // Emit the value as an integer.
       APInt AsInt = Value.bitcastToAPInt();
       getStreamer().EmitIntValue(AsInt.getLimitedValue(),
                                  AsInt.getBitWidth() / 8);
 
-      if (getLexer().is(AsmToken::EndOfStatement))
+      if (Lexer.is(AsmToken::EndOfStatement))
         break;
 
-      if (getLexer().isNot(AsmToken::Comma))
+      if (Lexer.isNot(AsmToken::Comma))
         return TokError("unexpected token in directive");
       Lex();
     }
   }
 
   Lex();
   return false;
 }
while (getLexer().isNot(AsmToken::EndOfStatement)) {
     }
 
     Parameters.push_back(std::move(Parameter));
 
     if (getLexer().is(AsmToken::Comma))
       Lex();
   }
 
-  // Eat the end of statement.
-  Lex();
+  // Eat just the end of statement.
+  Lexer.Lex();
 
+  // Consuming deferred text, so use Lexer.Lex to ignore Lexing Errors
   AsmToken EndToken, StartToken = getTok();
   unsigned MacroDepth = 0;
-
   // Lex the macro definition.
   for (;;) {
+    // Ignore Lexing errors in macros.
+    while (Lexer.is(AsmToken::Error)) {
+      Lexer.Lex();
+    }
+
     // Check whether we have reached the end of the file.
     if (getLexer().is(AsmToken::Eof))
       return Error(DirectiveLoc, "no matching '.endmacro' in definition");
 
     // Otherwise, check whether we have reach the .endmacro.
     if (getLexer().is(AsmToken::Identifier)) {
       if (getTok().getIdentifier() == ".endm" ||
           getTok().getIdentifier() == ".endmacro") {
         if (MacroDepth == 0) { // Outermost macro.
           EndToken = getTok();
-          Lex();
+          Lexer.Lex();
           if (getLexer().isNot(AsmToken::EndOfStatement))
             return TokError("unexpected token in '" + EndToken.getIdentifier() +
                             "' directive");
           break;
         } else {
           // Otherwise we just found the end of an inner macro.
           --MacroDepth;
         }
static bool isSymbolUsedInExpression(const MCSymbol *Sym, const MCExpr *Value) {
   }
 
   llvm_unreachable("Unknown expr kind!");
 }
 
 bool parseAssignmentExpression(StringRef Name, bool allow_redef,
                                MCAsmParser &Parser, MCSymbol *&Sym,
                                const MCExpr *&Value) {
-  MCAsmLexer &Lexer = Parser.getLexer();
 
   // FIXME: Use better location, we should use proper tokens.
-  SMLoc EqualLoc = Lexer.getLoc();
+  SMLoc EqualLoc = Parser.getTok().getLoc();
 
   if (Parser.parseExpression(Value)) {
     Parser.TokError("missing expression");
     Parser.eatToEndOfStatement();
     return true;
   }
 
   // Note: we don't count b as used in "a = b". This is to allow
   // a = b
   // b = c
 
-  if (Lexer.isNot(AsmToken::EndOfStatement))
+  if (Parser.getTok().isNot(AsmToken::EndOfStatement))
     return Parser.TokError("unexpected token in assignment");
 
   // Eat the end of statement marker.
   Parser.Lex();
 
   // Validate that the LHS is allowed to be a variable (either it has not been
   // used as a symbol, or it is an absolute symbol).
   Sym = Parser.getContext().lookupSymbol(Name);

lib/MC/MCParser/ELFAsmParser.cpp

 
 bool ELFAsmParser::ParseSectionSwitch(StringRef Section, unsigned Type,
                                       unsigned Flags, SectionKind Kind) {
   const MCExpr *Subsection = nullptr;
   if (getLexer().isNot(AsmToken::EndOfStatement)) {
     if (getParser().parseExpression(Subsection))
       return true;
   }
+  Lex();
 
   getStreamer().SwitchSection(getContext().getELFSection(Section, Type, Flags),
                               Subsection);
 
   return false;
 }
 
 bool ELFAsmParser::ParseDirectiveSize(StringRef, SMLoc) {

lib/MC/MCParser/MCAsmLexer.cpp

 //===-- MCAsmLexer.cpp - Abstract Asm Lexer Interface ---------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/MC/MCParser/MCAsmLexer.h"
 #include "llvm/Support/SourceMgr.h"
 
 using namespace llvm;
 
 MCAsmLexer::MCAsmLexer() : TokStart(nullptr), SkipSpace(true) {
-  CurTok.emplace_back(AsmToken::Error, StringRef());
+  CurTok.emplace_back(AsmToken::Space, StringRef());
 }
 
 MCAsmLexer::~MCAsmLexer() {
 }
 
 SMLoc MCAsmLexer::getLoc() const {
   return SMLoc::getFromPointer(TokStart);
 }

lib/MC/MCStreamer.cpp

 
 raw_ostream &MCStreamer::GetCommentOS() {
   // By default, discard comments.
   return nulls();
 }
 
 void MCStreamer::emitRawComment(const Twine &T, bool TabPrefix) {}
 
+void MCStreamer::addExplicitComment(const Twine &T) {}
+void MCStreamer::outputExplicitComments() {}
+
 void MCStreamer::generateCompactUnwindEncodings(MCAsmBackend *MAB) {
   for (auto &FI : DwarfFrameInfos)
     FI.CompactUnwindEncoding =
         (MAB ? MAB->generateCompactUnwindEncoding(FI.Instructions) : 0);
 }
 
 /// EmitIntValue - Special case of EmitValue that avoids the client having to
 /// pass in a MCExpr for constant integers.

lib/Target/ARM/AsmParser/ARMAsmParser.cpp

bool ARMAsmParser::parseDirectiveObjectArch(SMLoc L) {
   if (getLexer().isNot(AsmToken::Identifier)) {
     Error(getLexer().getLoc(), "unexpected token");
     Parser.eatToEndOfStatement();
     return false;
   }
 
   StringRef Arch = Parser.getTok().getString();
   SMLoc ArchLoc = Parser.getTok().getLoc();
-  getLexer().Lex();
+  Lex();
 
   unsigned ID = ARM::parseArch(Arch);
 
   if (ID == ARM::AK_INVALID) {
     Error(ArchLoc, "unknown architecture '" + Arch + "'");
     Parser.eatToEndOfStatement();
     return false;
   }
bool ARMAsmParser::parseDirectiveArchExtension(SMLoc L) {
   if (getLexer().isNot(AsmToken::Identifier)) {
     Error(getLexer().getLoc(), "unexpected token");
     Parser.eatToEndOfStatement();
     return false;
   }
 
   StringRef Name = Parser.getTok().getString();
   SMLoc ExtLoc = Parser.getTok().getLoc();
-  getLexer().Lex();
+  Lex();
 
   bool EnableFeature = true;
   if (Name.startswith_lower("no")) {
     EnableFeature = false;
     Name = Name.substr(2);
   }
   unsigned FeatureKind = ARM::parseArchExt(Name);
   if (FeatureKind == ARM::AEK_INVALID)

lib/Target/Hexagon/AsmParser/HexagonAsmParser.cpp

if (CheckOk) {
     }
   }
 
   return false; // No error
 }
 
 bool HexagonAsmParser::matchBundleOptions() {
   MCAsmParser &Parser = getParser();
-  MCAsmLexer &Lexer = getLexer();
   while (true) {
     if (!Parser.getTok().is(AsmToken::Colon))
       return false;
-    Lexer.Lex();
+    Lex();
     StringRef Option = Parser.getTok().getString();
     if (Option.compare_lower("endloop0") == 0)
       HexagonMCInstrInfo::setInnerLoop(MCB);
     else if (Option.compare_lower("endloop1") == 0)
       HexagonMCInstrInfo::setOuterLoop(MCB);
     else if (Option.compare_lower("mem_noshuf") == 0)
       HexagonMCInstrInfo::setMemReorderDisabled(MCB);
     else if (Option.compare_lower("mem_shuf") == 0)
       HexagonMCInstrInfo::setMemStoreReorderEnabled(MCB);
     else
       return true;
-    Lexer.Lex();
+    Lex();
   }
 }
 
 // For instruction aliases, immediates are generated rather than
 // MCConstantExpr.  Convert them for uniform MCExpr.
 // Also check for signed/unsigned mismatches and warn
 void HexagonAsmParser::canonicalizeImmediates(MCInst &MCI) {
   MCInst NewInst;
return previousEqual(Operands, Index, "loop0") ||
          previousEqual(Operands, Index, "sp3loop0");
 }
 }
 
 bool HexagonAsmParser::splitIdentifier(OperandVector &Operands) {
   AsmToken const &Token = getParser().getTok();
   StringRef String = Token.getString();
   SMLoc Loc = Token.getLoc();
-  getLexer().Lex();
+  Lex();
   do {
     std::pair<StringRef, StringRef> HeadTail = String.split('.');
     if (!HeadTail.first.empty())
       Operands.push_back(HexagonOperand::CreateToken(HeadTail.first, Loc));
     if (!HeadTail.second.empty())
       Operands.push_back(HexagonOperand::CreateToken(
           String.substr(HeadTail.first.size(), 1), Loc));
     String = HeadTail.second;

 
 bool HexagonAsmParser::parseExpression(MCExpr const *& Expr) {
   llvm::SmallVector<AsmToken, 4> Tokens;
   MCAsmLexer &Lexer = getLexer();
   bool Done = false;
   static char const * Comma = ",";
   do {
     Tokens.emplace_back (Lexer.getTok());
-    Lexer.Lex();
+    Lex();
     switch (Tokens.back().getKind())
     {
     case AsmToken::TokenKind::Hash:
       if (Tokens.size () > 1)
         if ((Tokens.end () - 2)->getKind() == AsmToken::TokenKind::Plus) {
           Tokens.insert(Tokens.end() - 2,
                         AsmToken(AsmToken::TokenKind::Comma, Comma));
           Done = true;

 /// Parse an instruction.
 bool HexagonAsmParser::parseInstruction(OperandVector &Operands) {
   MCAsmParser &Parser = getParser();
   MCAsmLexer &Lexer = getLexer();
   while (true) {
     AsmToken const &Token = Parser.getTok();
     switch (Token.getKind()) {
     case AsmToken::EndOfStatement: {
-      Lexer.Lex();
+      Lex();
       return false;
     }
     case AsmToken::LCurly: {
       if (!Operands.empty())
         return true;
       Operands.push_back(
           HexagonOperand::CreateToken(Token.getString(), Token.getLoc()));
-      Lexer.Lex();
+      Lex();
       return false;
     }
     case AsmToken::RCurly: {
       if (Operands.empty()) {
         Operands.push_back(
             HexagonOperand::CreateToken(Token.getString(), Token.getLoc()));
-        Lexer.Lex();
+        Lex();
       }
       return false;
     }
     case AsmToken::Comma: {
-      Lexer.Lex();
+      Lex();
       continue;
     }
     case AsmToken::EqualEqual:
     case AsmToken::ExclaimEqual:
     case AsmToken::GreaterEqual:
     case AsmToken::GreaterGreater:
     case AsmToken::LessEqual:
     case AsmToken::LessLess: {
       Operands.push_back(HexagonOperand::CreateToken(
           Token.getString().substr(0, 1), Token.getLoc()));
       Operands.push_back(HexagonOperand::CreateToken(
           Token.getString().substr(1, 1), Token.getLoc()));
-      Lexer.Lex();
+      Lex();
       continue;
     }
     case AsmToken::Hash: {
       bool MustNotExtend = false;
       bool ImplicitExpression = implicitExpressionLocation(Operands);
       SMLoc ExprLoc = Lexer.getLoc();
       if (!ImplicitExpression)
         Operands.push_back(
           HexagonOperand::CreateToken(Token.getString(), Token.getLoc()));
-      Lexer.Lex();
+      Lex();
       bool MustExtend = false;
       bool HiOnly = false;
       bool LoOnly = false;
       if (Lexer.is(AsmToken::Hash)) {
-        Lexer.Lex();
+        Lex();
         MustExtend = true;
       } else if (ImplicitExpression)
         MustNotExtend = true;
       AsmToken const &Token = Parser.getTok();
       if (Token.is(AsmToken::Identifier)) {
         StringRef String = Token.getString();
         if (String.lower() == "hi") {
           HiOnly = true;
         } else if (String.lower() == "lo") {
           LoOnly = true;
         }
         if (HiOnly || LoOnly) {
           AsmToken LParen = Lexer.peekTok();
           if (!LParen.is(AsmToken::LParen)) {
             HiOnly = false;
             LoOnly = false;
           } else {
-            Lexer.Lex();
+            Lex();
           }
         }
       }
       MCExpr const *Expr = nullptr;
       if (parseExpression(Expr))
         return true;
       int64_t Value;
       MCContext &Context = Parser.getContext();

lib/Target/Lanai/AsmParser/LanaiAsmParser.cpp

if (Lexer.is(AsmToken::EndOfStatement) && Name.startswith("bt") &&
       Operands.size() == 3) {
     Operands.erase(Operands.begin(), Operands.begin() + 2);
     Operands.insert(Operands.begin(), LanaiOperand::CreateToken("bt", NameLoc));
   }
 
   // Parse until end of statement, consuming commas between operands
   while (Lexer.isNot(AsmToken::EndOfStatement) && Lexer.is(AsmToken::Comma)) {
     // Consume comma token
-    Lexer.Lex();
+    Lex();
 
     // Parse next operand
     if (parseOperand(&Operands, Mnemonic) != MatchOperand_Success)
       return true;
   }
 
   if (IsMemoryAssignmentError(Operands)) {
     Error(Parser.getTok().getLoc(),

lib/Target/PowerPC/AsmParser/PPCAsmParser.cpp

bool PPCAsmParser::ParseInstruction(ParseInstructionInfo &Info, StringRef Name,
 
   // Parse the first operand
   if (ParseOperand(Operands))
     return true;
 
   while (getLexer().isNot(AsmToken::EndOfStatement) &&
          getLexer().is(AsmToken::Comma)) {
     // Consume the comma token
-    getLexer().Lex();
+    Lex();
 
     // Parse the next operand
     if (ParseOperand(Operands))
       return true;
   }
 
   // We'll now deal with an unfortunate special case: the syntax for the dcbt
   // and dcbtst instructions differs for server vs. embedded cores.

test/MC/AsmParser/floating-literals.s

 # CHECK: .long 1094713344
 .float 0x3p2
 # CHECK: .long 872284160
 .float 0x7fp-30
 # CHECK: .long 3212836864
 .float -0x1.0p0
 
 # CHECK-ERROR: invalid hexadecimal floating-point constant: expected at least one exponent digit
-# CHECK-ERROR: unexpected token in directive
 .float 0xa.apa
 
 # CHECK-ERROR: invalid hexadecimal floating-point constant: expected at least one exponent digit
-# CHECK-ERROR: unexpected token in directive
 .double -0x1.2p+
 
 # CHECK-ERROR: invalid hexadecimal floating-point constant: expected at least one exponent digit
-# CHECK-ERROR: unexpected token in directive
 .double -0x1.2p
 
 # CHECK-ERROR: invalid hexadecimal floating-point constant: expected at least one significand digit
-# CHECK-ERROR: unexpected token in directive
 .float 0xp2
 
 # CHECK-ERROR: invalid hexadecimal floating-point constant: expected at least one significand digit
-# CHECK-ERROR: unexpected token in directive
 .float 0x.p5
 
 # CHECK-ERROR: error: invalid hexadecimal floating-point constant: expected exponent part 'p'
-# CHECK-ERROR: unexpected token in directive
 .float 0x1.2

test/MC/AsmParser/hash-directive.s

+# RUN: not llvm-mc -triple i386-unknown-unknown %s 2>&1 | FileCheck %s
+error 
+# CHECK: hash-directive.s:[[@LINE-1]]:1: error  
+# 3 "FILE1" 1 #<- This is a CPP Hash w/ comment
+error
+# CHECK: FILE1:3:1: error
+# 0 "" 2 #<- This is too
+error 
+# CHECK: hash-directive.s:[[@LINE-1]]:1: error
+ # 1 "FILE2" 2 #<- This is a comment
+error 
+# CHECK: hash-directive.s:[[@LINE-1]]:1: error  
+nop; # 6 "FILE3" 2 #<- This is a still comment
+error
+# CHECK: hash-directive.s:[[@LINE-1]]:1: error
+nop;# 6 "FILE4" 2
+  nop;
+error
+# CHECK: FILE4:7:1: error
+# 0 "" 2
+/*comment*/# 6 "FILE5" 2 #<- This is a comment
+error
+# CHECK: hash-directive.s:[[@LINE-1]]:1: error  

test/MC/AsmParser/inline-comments.ll

+; RUN: llc %s -o - | sed -n -e '/#APP/,/#NO_APP/p' > %t
+; RUN: sed -n -e 's/^;CHECK://p' %s > %t2
+; RUN: diff %t %t2
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; Function Attrs: nounwind uwtable
+define void @foo() #0 {
+entry:
+  call void asm sideeffect "", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: #NO_APP
+  call void asm sideeffect " ", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK:
+;CHECK: #NO_APP
+  call void asm sideeffect "\0A", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK:
+;CHECK:
+;CHECK: #NO_APP
+  call void asm sideeffect "/*isolated c comment*/", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: #isolated c comment
+;CHECK: #NO_APP
+  call void asm sideeffect "/**/", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: #
+;CHECK: #NO_APP
+  call void asm sideeffect "/*comment with\0Anewline*/", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: #comment with#newline
+;CHECK: #NO_APP
+  call void asm sideeffect "//isolated line comment", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: #isolated line comment
+;CHECK: #NO_APP
+  call void asm sideeffect "#isolated line comment", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: #isolated line comment
+;CHECK: #NO_APP
+   call void asm sideeffect "nop /* after nop */", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: nop # after nop 
+;CHECK: #NO_APP
+  call void asm sideeffect "nop // after nop", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: nop # after nop
+;CHECK: #NO_APP
+  call void asm sideeffect "nop # after nop", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: nop # after nop
+;CHECK: #NO_APP
+  call void asm sideeffect "nop /* after explicit ended nop */", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: nop # after explicit ended nop 
+;CHECK: #NO_APP
+  call void asm sideeffect "nop # after explicit ended nop", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: nop # after explicit ended nop
+;CHECK: #NO_APP
+  call void asm sideeffect "nop # after explicit end nop", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: nop # after explicit end nop
+;CHECK: #NO_APP
+  call void asm sideeffect "/* before nop */ nop", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: nop # before nop 
+;CHECK: #NO_APP
+  call void asm sideeffect "//comment with escaped newline\0A", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: #comment with escaped newline
+;CHECK:
+;CHECK: #NO_APP
+  call void asm sideeffect "/*0*/xor/*1*/%eax,/*2*/%ecx/*3*///eol", "~{dirflag},~{fpsr},~{flags}"() #0
+;CHECK: #APP
+;CHECK: xorl  %eax, %ecx  #0#1#2#3#eol
+;CHECK: #NO_APP
+  ret void
+}
+
+attributes #0 = { nounwind }
+
+!llvm.ident = !{!0}
+
+!0 = !{!"clang version 3.9.0 (trunk 268625) (llvm/trunk 268631)"}

test/MC/AsmParser/macro_parsing.s

+# RUN: llvm-mc -triple i386-unknown-unknown %s | FileCheck %s
+
+        .macro DEF num
+        int $0x\num
+        .endm
+        DEF 02
+        DEF 08
+        DEF 09
+        DEF 0A
+        DEF 10
+
+# CHECK: int $2
+# CHECK: int $8
+# CHECK: int $9
+# CHECK: int $10
+# CHECK: int $16

test/MC/AsmParser/preserve-comments.s

+  #RUN: llvm-mc -preserve-comments -n -triple i386-linux-gnu < %s > %t
+  #RUN: diff %s %t
+  .text
+
+  nop
+  #if DIRECTIVE COMMENT
+  ## WHOLE LINE COMMENT
+  cmpl  $196, %eax  ## EOL COMMENT
+  #endif

tools/llc/llc.cpp

 TimeCompilations("time-compilations", cl::Hidden, cl::init(1u),
                  cl::value_desc("N"),
                  cl::desc("Repeat compilation N times for timing"));
 
 static cl::opt<bool>
 NoIntegratedAssembler("no-integrated-as", cl::Hidden,
                       cl::desc("Disable integrated assembler"));
 
+static cl::opt<bool>
+    NoPreserveComments("fno-preserve-as-comments", cl::Hidden,
+                       cl::desc("Preserve Comments in outputted assembly"));
+
 // Determine optimization level.
 static cl::opt<char>
 OptLevel("O",
          cl::desc("Optimization level. [-O0, -O1, -O2, or -O3] "
                   "(default = '-O2')"),
          cl::Prefix,
          cl::ZeroOrMore,
          cl::init(' '));
static int compileModule(char **argv, LLVMContext &Context) {
   case '0': OLvl = CodeGenOpt::None; break;
   case '1': OLvl = CodeGenOpt::Less; break;
   case '2': OLvl = CodeGenOpt::Default; break;
   case '3': OLvl = CodeGenOpt::Aggressive; break;
   }
 
   TargetOptions Options = InitTargetOptionsFromCodeGenFlags();
   Options.DisableIntegratedAS = NoIntegratedAssembler;
+  Options.PreserveAsmComments = !NoPreserveComments;
   Options.MCOptions.ShowMCEncoding = ShowMCEncoding;
   Options.MCOptions.MCUseDwarfDirectory = EnableDwarfDirectory;
   Options.MCOptions.AsmVerbose = AsmVerbose;
 
   std::unique_ptr<TargetMachine> Target(
       TheTarget->createTargetMachine(TheTriple.getTriple(), CPUStr, FeaturesStr,
                                      Options, getRelocModel(), CMModel, OLvl));
 

tools/llvm-mc/llvm-mc.cpp

 
 static cl::opt<bool>
 PrintImmHex("print-imm-hex", cl::init(false),
             cl::desc("Prefer hex format for immediate values"));
 
 static cl::list<std::string>
 DefineSymbol("defsym", cl::desc("Defines a symbol to be an integer constant"));
 
+static cl::opt<bool>
+    PreserveComments("preserve-comments",
+                     cl::desc("Preserve Comments in outputted assembly"));
+
 enum OutputFileType {
   OFT_Null,
   OFT_AssemblyFile,
   OFT_ObjectFile
 };
 static cl::opt<OutputFileType>
 FileType("filetype", cl::init(OFT_AssemblyFile),
   cl::desc("Choose an output file type:"),
int main(int argc, char **argv) {
   if (CompressDebugSections != DebugCompressionType::DCT_None) {
     if (!zlib::isAvailable()) {
       errs() << ProgName
              << ": build tools with zlib to enable -compress-debug-sections";
       return 1;
     }
     MAI->setCompressDebugSections(CompressDebugSections);
   }
+  MAI->setPreserveAsmComments(PreserveComments);
 
   // FIXME: This is not pretty. MCContext has a ptr to MCObjectFileInfo and
   // MCObjectFileInfo needs a MCContext reference in order to initialize itself.
   MCObjectFileInfo MOFI;
   MCContext Ctx(MAI.get(), MRI.get(), &MOFI, &SrcMgr);
   MOFI.InitMCObjectFileInfo(TheTriple, PIC, CMModel, Ctx);
 
   if (SaveTempLabels)