[ms] [llvm-ml] Support built-in text macros

[epastor]

Summary

Add support for all built-in text macros supported by ML64:
@Date, @Time, @FileName, @FileCur, and @CurSeg.

[epastor]

Rebase on parent

[epastor]

Rebase on parent

[thakis]

llvm/lib/MC/MCParser/MasmParser.cpp

1175

Why do you do both lookups? Inserting into Variables with a built-in name errors, so we know that if it's in BuiltinSymbolMap, it can't be in Variables. So do

auto BuiltinIt = ...
if (BuiltinIt != end()) {
  ...
} else {
  auto VarIt = ...;
  if (VarIt != end() ... )
    ...
}

3643

Like above, looking up both in this loop looks weird to me. I'd structure this like

while (true) {
  // Lookup builtinText, check if not end, handle builtin case, continue
  // Same for variables
  break;
}

7688

Please add a flag to set the the timestamp on the commandline, for deterministic build reasons. lld-link has /timestamp:, so I guess let's add that here too. (And then set TT to the arg of /timestamp: if it's passed and to time(nullptr) else)

7689

Hm, this makes this function non-thread-safe. I think that's fine, but maybe add a // Not thread-safe at the end of the line to make that explicit.

7692

Instead of lines 7701-7705, do this:

char TmpBuffer[sizeof("mm/dd/yy")];
strftime(TmpBuffer, sizeof(TmpBuffer), "%D", TM);
return TmpBuffer;

Don't even need a dedicated variable for TM then.

7697

24-hour clock I guess? Not clear from the comment.

7698

/timestamp: support there too

7700

similar to above:

char TmpBuffer[sizeof("HH:MM:SS")];
strftime(TmpBuffer, sizeof(TmpBuffer), "%T", TM);
return TmpBuffer;

7713

sys::path::stem() already extracts the filename, no need to call sys::path::filename() too. This is not supposed to include the extension, yes?

7716

Do we have to do the upper()?

llvm/test/tools/llvm-ml/builtin_symbols.asm

43

Instead of the previous two lines, you could do CHECK: FileCur = {{.*}}builtin_symbols_t5.inc Could even use {{.*[\\/]}} to check that there's a directory separator in front of builtin_symbols

53

with /timestamp: you could put actual numbers here :)

[epastor]

Add -timestamp and -utc flags to support deterministic builds

[epastor]

llvm/lib/MC/MCParser/MasmParser.cpp

1175

There seemed to be a lot of work going on in LLVM to avoid excessively-nested if statements, so I thought this was a worthwhile tradeoff. Thanks for correcting! Fixed.

7688

I was adding that in a followup commit, but I've merged it in here now.

7689

Switched to localtime_r; I don't think localtime has any real advantage here.

7697

Clarified.

7713

Interesting, I didn't realize stem did both. Fixed!

7716

If we want to match ML.EXE, yes - it always uses an upper-cased form of the stem for the @Filename built-in.

llvm/test/tools/llvm-ml/builtin_symbols.asm

43

Good point; cleaned up.

53

I had to add another flag for this, because MASM canonically uses the machine's local timezone for its built-ins; -utc now forces the use of GMT for the timezone.

[epastor]

Rebase on parent

[thakis]

Looks good now except for below.

llvm/lib/MC/MCParser/MasmParser.cpp

7689

Is that available on all platforms? If so, ok.

Does "%D" for format string not work?

7695

Does "%T" not work as format string?

[epastor]

Switch back to gmtime/localtime (not thread-safe, but in a generally-safe location - and more cross-platform)

Also address strftime comments.

llvm/lib/MC/MCParser/MasmParser.cpp

 //===- AsmParser.cpp - Parser for Assembly Files --------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This class implements the parser for assembly files.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"

 #include "llvm/MC/MCValue.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/MD5.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Support/SMLoc.h"
 #include "llvm/Support/SourceMgr.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cassert>
 #include <cctype>
 #include <climits>
 #include <cstddef>
 #include <cstdint>
+#include <ctime>
 #include <deque>
 #include <memory>
 #include <sstream>
 #include <string>
 #include <tuple>
 #include <utility>
 #include <vector>
 
private:
   };
 
   /// Maps builtin name --> BuiltinSymbol enum, for builtins handled by this
   /// class.
   StringMap<BuiltinSymbol> BuiltinSymbolMap;
 
   const MCExpr *evaluateBuiltinValue(BuiltinSymbol Symbol, SMLoc StartLoc);
 
+  llvm::Optional<std::string> evaluateBuiltinTextMacro(BuiltinSymbol Symbol,
+                                                       SMLoc StartLoc);
+
   // ".ascii", ".asciz", ".string"
   bool parseDirectiveAscii(StringRef IDVal, bool ZeroTerminated);
 
   // "byte", "word", ...
   bool emitIntValue(const MCExpr *Value, unsigned Size);
   bool parseScalarInitializer(unsigned Size,
                               SmallVectorImpl<const MCExpr *> &Values,
                               unsigned StringPadLength = 0);
void MasmParser::jumpToLoc(SMLoc Loc, unsigned InBuffer,
                            bool EndStatementAtEOF) {
   CurBuffer = InBuffer ? InBuffer : SrcMgr.FindBufferContainingLoc(Loc);
   Lexer.setBuffer(SrcMgr.getMemoryBuffer(CurBuffer)->getBuffer(),
                   Loc.getPointer(), EndStatementAtEOF);
 }
 
 bool MasmParser::expandCurrentToken() {
   const AsmToken *tok = &getTok();
-  auto it = Variables.find(tok->getIdentifier().lower());
-  const llvm::MCAsmMacro *M =
-      getContext().lookupMacro(tok->getIdentifier().lower());
-  if (it != Variables.end() && it->second.IsText) {
-    std::unique_ptr<MemoryBuffer> Instantiation =
-        MemoryBuffer::getMemBufferCopy(it->second.TextValue, "<instantiation>");
-
-    // Jump to the macro instantiation and prime the lexer.
-    CurBuffer = SrcMgr.AddNewSourceBuffer(std::move(Instantiation),
-                                          getTok().getEndLoc());
-    Lexer.setBuffer(SrcMgr.getMemoryBuffer(CurBuffer)->getBuffer(), nullptr,
-                    /*EndStatementAtEOF=*/false);
-    EndStatementAtEOFStack.push_back(false);
-    Lexer.Lex();
-    return false;
-  } else if (M && M->IsFunction && peekTok().is(AsmToken::LParen)) {
+  const std::string IDLower = tok->getIdentifier().lower();
+  const llvm::MCAsmMacro *M = getContext().lookupMacro(IDLower);
+  if (M && M->IsFunction && peekTok().is(AsmToken::LParen)) {
     // This is a macro function invocation; expand it in place.
     const AsmToken MacroTok = *tok;
     Lexer.Lex();
     if (handleMacroInvocation(M, MacroTok.getLoc())) {
       Lexer.UnLex(AsmToken(AsmToken::Error, MacroTok.getIdentifier()));
       Lexer.Lex();
     }
     return false;
   }
+
+  llvm::Optional<std::string> ExpandedValue;
+  auto BuiltinIt = BuiltinSymbolMap.find(IDLower);
+  auto VarIt = Variables.find(IDLower);
+  if (BuiltinIt != BuiltinSymbolMap.end()) {
+    ExpandedValue =
+        evaluateBuiltinTextMacro(BuiltinIt->getValue(), tok->getLoc());
+  } else if (VarIt != Variables.end() && VarIt->getValue().IsText) {
+    ExpandedValue = VarIt->getValue().TextValue;
+  }
+
+  if (!ExpandedValue.hasValue())
     return true;
+  std::unique_ptr<MemoryBuffer> Instantiation =
+      MemoryBuffer::getMemBufferCopy(*ExpandedValue, "<instantiation>");
+
+  // Jump to the macro instantiation and prime the lexer.
+  CurBuffer =
+      SrcMgr.AddNewSourceBuffer(std::move(Instantiation), getTok().getEndLoc());
+  Lexer.setBuffer(SrcMgr.getMemoryBuffer(CurBuffer)->getBuffer(), nullptr,
+                  /*EndStatementAtEOF=*/false);
+  EndStatementAtEOFStack.push_back(false);
+  Lexer.Lex();
+  return false;
 }
 
 const AsmToken &MasmParser::Lex(bool ExpandNextToken) {
   if (Lexer.getTok().is(AsmToken::Error))
     Error(Lexer.getErrLoc(), Lexer.getErr());
 
   // if it's a end of statement with a comment in it
   if (getTok().is(AsmToken::EndOfStatement)) {
while (!Body.empty()) {
     unsigned I = Pos;
     bool InitialAmpersand = (Body[I] == '&');
     if (InitialAmpersand) {
       ++I;
       ++Pos;
     }
     while (I < End && isMacroParameterChar(Body[I]))
       ++I;
 
     const char *Begin = Body.data() + Pos;
     StringRef Argument(Begin, I - Pos);
+    const std::string ArgumentLower = Argument.lower();
     unsigned Index = 0;
 
     for (; Index < NParameters; ++Index)
-      if (Parameters[Index].Name == Argument)
+      if (Parameters[Index].Name.equals_insensitive(ArgumentLower))
         break;
 
     if (Index == NParameters) {
       if (InitialAmpersand)
         OS << '&';
-      auto it = LocalSymbols.find(Argument.lower());
+      auto it = LocalSymbols.find(ArgumentLower);
       if (it != LocalSymbols.end())
         OS << it->second;
       else
         OS << Argument;
       Pos = I;
     } else {
       for (const AsmToken &Token : A[Index]) {
         // In MASM, you can write '%expr'.
if (!Expr->evaluateAsAbsolute(Value, getStreamer().getAssemblerPtr())) {
     Var.Redefinable = Variable::REDEFINABLE;
 
     return false;
   }
 
   MCSymbol *Sym = getContext().getOrCreateSymbol(Var.Name);
 
   const MCConstantExpr *PrevValue =
-      Sym->isVariable()
-          ? dyn_cast_or_null<MCConstantExpr>(Sym->getVariableValue())
+      Sym->isVariable() ? dyn_cast_or_null<MCConstantExpr>(
+                              Sym->getVariableValue(/*SetUsed*/ false))
                         : nullptr;
   if (Var.IsText || !PrevValue || PrevValue->getValue() != Value) {
     switch (Var.Redefinable) {
     case Variable::NOT_REDEFINABLE:
       return Error(getTok().getLoc(), "invalid variable redefinition");
     case Variable::WARN_ON_REDEFINITION:
       if (Warning(NameLoc, "redefining '" + Name +
                                "', already defined on the command line")) {
         return true;
bool MasmParser::parseTextItem(std::string &Data) {
   case AsmToken::Less:
   case AsmToken::LessEqual:
   case AsmToken::LessLess:
   case AsmToken::LessGreater:
     return parseAngleBracketString(Data);
   case AsmToken::Identifier: {
     // This must be a text macro; we need to expand it accordingly.
     StringRef ID;
+    SMLoc StartLoc = getTok().getLoc();
     if (parseIdentifier(ID))
       return true;
     Data = ID.str();
 
-    auto it = Variables.find(ID.lower());
-    if (it == Variables.end()) {
-      // Not a variable; since we haven't used the token, put it back for better
-      // error recovery.
+    auto BuiltinIt = BuiltinSymbolMap.find(ID.lower());
+    auto VarIt = Variables.find(ID.lower());
+    if (BuiltinIt == BuiltinSymbolMap.end() && VarIt == Variables.end()) {
+      // Not a variable or built-in text macro; since we haven't used the token,
+      // put it back for better error recovery.
       getLexer().UnLex(AsmToken(AsmToken::Identifier, ID));
       return true;
     }
 
-    while (it != Variables.end()) {
-      const Variable &Var = it->second;
+    while (BuiltinIt != BuiltinSymbolMap.end() || VarIt != Variables.end()) {
+      if (BuiltinIt != BuiltinSymbolMap.end()) {
+        llvm::Optional<std::string> BuiltinText =
+            evaluateBuiltinTextMacro(BuiltinIt->second, StartLoc);
+        if (!BuiltinText.hasValue()) {
+          // Not a text macro; not usable in TextItem context. Since we haven't
+          // used the token, put it back for better error recovery.
+          getLexer().UnLex(AsmToken(AsmToken::Identifier, ID));
+          return true;
+        }
+        Data = BuiltinText.getValue();
+      } else {
+        const Variable &Var = VarIt->second;
         if (!Var.IsText) {
           // Not a text macro; not usable in TextItem context. Since we haven't
           // used the token, put it back for better error recovery.
           getLexer().UnLex(AsmToken(AsmToken::Identifier, ID));
           return true;
         }
         Data = Var.TextValue;
-      it = Variables.find(StringRef(Data).lower());
+      }
+      VarIt = Variables.find(StringRef(Data).lower());
+      BuiltinIt = BuiltinSymbolMap.find(StringRef(Data).lower());
     }
     return false;
   }
   }
   llvm_unreachable("unhandled token kind");
 }
 
 /// parseDirectiveAscii:

 }
 
 bool MasmParser::expandStatement(SMLoc Loc) {
   std::string Body = parseStringTo(AsmToken::EndOfStatement);
   SMLoc EndLoc = getTok().getLoc();
 
   MCAsmMacroParameters Parameters;
   MCAsmMacroArguments Arguments;
+
+  StringMap<std::string> BuiltinValues;
+  for (const auto &S : BuiltinSymbolMap) {
+    const BuiltinSymbol &Sym = S.getValue();
+    if (llvm::Optional<std::string> Text = evaluateBuiltinTextMacro(Sym, Loc)) {
+      BuiltinValues[S.getKey().lower()] = std::move(*Text);
+    }
+  }
+  for (const auto &B : BuiltinValues) {
+    MCAsmMacroParameter P;
+    MCAsmMacroArgument A;
+    P.Name = B.getKey();
+    P.Required = true;
+    A.push_back(AsmToken(AsmToken::String, B.getValue()));
+
+    Parameters.push_back(std::move(P));
+    Arguments.push_back(std::move(A));
+  }
+
   for (const auto &V : Variables) {
     const Variable &Var = V.getValue();
     if (Var.IsText) {
-      Parameters.emplace_back();
-      Arguments.emplace_back();
-      MCAsmMacroParameter &P = Parameters.back();
-      MCAsmMacroArgument &A = Arguments.back();
+      MCAsmMacroParameter P;
+      MCAsmMacroArgument A;
       P.Name = Var.Name;
       P.Required = true;
       A.push_back(AsmToken(AsmToken::String, Var.TextValue));
+
+      Parameters.push_back(std::move(P));
+      Arguments.push_back(std::move(A));
     }
   }
   MacroLikeBodies.emplace_back(StringRef(), Body, Parameters);
   MCAsmMacro M = MacroLikeBodies.back();
 
   // Expand the statement in a new buffer.
   SmallString<80> Buf;
   raw_svector_ostream OS(Buf);

 }
 
 void MasmParser::initializeBuiltinSymbolMap() {
   // Numeric built-ins (supported in all versions)
   BuiltinSymbolMap["@version"] = BI_VERSION;
   BuiltinSymbolMap["@line"] = BI_LINE;
 
   // Text built-ins (supported in all versions)
-  // BuiltinSymbolMap["@date"] = BI_DATE;
-  // BuiltinSymbolMap["@time"] = BI_TIME;
-  // BuiltinSymbolMap["@filecur"] = BI_FILECUR;
-  // BuiltinSymbolMap["@filename"] = BI_FILENAME;
-  // BuiltinSymbolMap["@curseg"] = BI_CURSEG;
+  BuiltinSymbolMap["@date"] = BI_DATE;
+  BuiltinSymbolMap["@time"] = BI_TIME;
+  BuiltinSymbolMap["@filecur"] = BI_FILECUR;
+  BuiltinSymbolMap["@filename"] = BI_FILENAME;
+  BuiltinSymbolMap["@curseg"] = BI_CURSEG;
 
   // Some built-ins exist only for MASM32 (32-bit x86)
   if (getContext().getSubtargetInfo()->getTargetTriple().getArch() ==
       Triple::x86) {
     // Numeric built-ins
     // BuiltinSymbolMap["@cpu"] = BI_CPU;
     // BuiltinSymbolMap["@interface"] = BI_INTERFACE;
     // BuiltinSymbolMap["@wordsize"] = BI_WORDSIZE;
else
       Line = SrcMgr.FindLineNumber(ActiveMacros.front()->InstantiationLoc,
                                    ActiveMacros.front()->ExitBuffer);
     return MCConstantExpr::create(Line, getContext());
   }
   }
   llvm_unreachable("unhandled built-in symbol");
 }
 
+llvm::Optional<std::string>
+MasmParser::evaluateBuiltinTextMacro(BuiltinSymbol Symbol, SMLoc StartLoc) {
+  switch (Symbol) {
+  default:
+    return {};
+  case BI_DATE: {
+    // Current local date, formatted MM/DD/YY
+    time_t TT = time(nullptr);
+    struct tm *TM = localtime(&TT);
+    SmallString<32> TmpBuffer;
+    llvm::raw_svector_ostream TmpStream(TmpBuffer);
+    TmpStream << llvm::format("%02d/%02d/%02d", TM->tm_mon + 1, TM->tm_mday,
+                              TM->tm_year % 100);
+    return TmpStream.str().str();
+  }
+  case BI_TIME: {
+    // Current local time, formatted HH:MM:SS
+    time_t TT = time(nullptr);
+    struct tm *TM = localtime(&TT);
+    SmallString<32> TmpBuffer;
+    llvm::raw_svector_ostream TmpStream(TmpBuffer);
+    TmpStream << llvm::format("%02d:%02d:%02d", TM->tm_hour, TM->tm_min,
+                              TM->tm_sec);
+    return TmpStream.str().str();
+  }
+  case BI_FILECUR:
+    return SrcMgr
+        .getMemoryBuffer(
+            ActiveMacros.empty() ? CurBuffer : ActiveMacros.front()->ExitBuffer)
+        ->getBufferIdentifier()
+        .str();
+  case BI_FILENAME:
+    return sys::path::stem(sys::path::filename(
+                               SrcMgr.getMemoryBuffer(SrcMgr.getMainFileID())
+                                   ->getBufferIdentifier()))
+        .upper();
+  case BI_CURSEG:
+    return getStreamer().getCurrentSectionOnly()->getName().str();
+  }
+  llvm_unreachable("unhandled built-in symbol");
+}
+
 /// Create an MCAsmParser instance.
 MCAsmParser *llvm::createMCMasmParser(SourceMgr &SM, MCContext &C,
                                       MCStreamer &Out, const MCAsmInfo &MAI,
                                       unsigned CB) {
   return new MasmParser(SM, C, Out, MAI, CB);
 }

llvm/test/tools/llvm-ml/builtin_symbols.asm

-; RUN: llvm-ml -filetype=s %s /Fo /dev/null 2>&1 | FileCheck %s
+; RUN: llvm-ml -filetype=s %s /I %S /Fo /dev/null 2>&1 | FileCheck %s
+
+.code
 
 version_val TEXTEQU %@Version
 
 ECHO t1:
 %ECHO @Version = version_val
 ; CHECK-LABEL: t1:
 ; CHECK-NEXT: 1427
 

 ECHO
 
 line_val TEXTEQU %@Line
 
 ECHO t4:
 %ECHO @Line = line_val
 ; CHECK-LABEL: t4:
 ; CHECK-NEXT: @Line = [[# @LINE - 5]]
+
+ECHO t5:
+include builtin_symbols_t5.inc
+; CHECK-LABEL: t5:
+; CHECK: FileCur =
+; CHECK-SAME: builtin_symbols_t5.inc
+; CHECK: FileName =
+; CHECK-SAME: BUILTIN_SYMBOLS
+; CHECK-NOT: T5
+
+ECHO t6:
+%ECHO Date = @Date
+%ECHO Time = @Time
+
+; CHECK-LABEL: t6:
+; CHECK: Date = {{([[:digit:]]{2}/[[:digit:]]{2}/[[:digit:]]{2})}}
+; CHECK-NOT: {{[[:digit:]]}}
+; CHECK: Time = {{([[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2})}}
+
+end

llvm/test/tools/llvm-ml/builtin_symbols_t5.inc

+%echo FileCur = @FileCur
+%echo FileName = @FileName