[Lex] Use line markers in preprocessed assembly predefines file

[john.brawn]

Summary

GNU line marker directives are not recognised when preprocessing assembly files, meaning they can't be used in the predefines file meaning macros defined on the command line are reported as being built-in.

Change this to permit line markers but only in the predefines file, so we can correctly report command line macros as coming from the command line.

[aaron.ballman]

LGTM, but please add a release note about the fix when landing.

clang/docs/ReleaseNotes.rst

- We now generate a diagnostic for signed integer overflow due to unary minus
   (`#31643 <https://github.com/llvm/llvm-project/issues/31643>`)
 - Clang now warns by default for C++20 and later about deprecated capture of
   ``this`` with a capture default of ``=``. This warning can be disabled with
   ``-Wno-deprecated-this-capture``.
 - Clang had failed to emit some ``-Wundefined-internal`` for members of a local
   class if that class was first introduced with a forward declaration.
 - Diagnostic notes and fix-its are now generated for ``ifunc``/``alias`` attributes
   which point to functions whose names are mangled.
+- Diagnostics relating to macros on the command line of a preprocessed assembly
+  file are now reported as coming from the file ``<command line>`` instead of
+  ``<built-in>``.
 
 Bug Fixes in This Version
 -------------------------
 
 - Fix crash when diagnosing incorrect usage of ``_Nullable`` involving alias
   templates.
   (`#60344 <https://github.com/llvm/llvm-project/issues/60344>`_)
 - Fix confusing warning message when ``/clang:-x`` is passed in ``clang-cl``

clang/lib/Frontend/InitPreprocessor.cpp

void clang::InitializePreprocessor(
     const PCHContainerReader &PCHContainerRdr,
     const FrontendOptions &FEOpts) {
   const LangOptions &LangOpts = PP.getLangOpts();
   std::string PredefineBuffer;
   PredefineBuffer.reserve(4080);
   llvm::raw_string_ostream Predefines(PredefineBuffer);
   MacroBuilder Builder(Predefines);
 
-  // Emit line markers for various builtin sections of the file.  We don't do
-  // this in asm preprocessor mode, because "# 4" is not a line marker directive
-  // in this mode.
-  if (!PP.getLangOpts().AsmPreprocessor)
-    Builder.append("# 1 \"<built-in>\" 3");
+  // Emit line markers for various builtin sections of the file. The 3 here
+  // marks <built-in> as being a system header, which suppresses warnings when
+  // the same macro is defined multiple times.
 
   // Install things like __POWERPC__, __GNUC__, etc into the macro table.
   if (InitOpts.UsePredefines) {
     // FIXME: This will create multiple definitions for most of the predefined
     // macros. This is not the right way to handle this.
     if ((LangOpts.CUDA || LangOpts.OpenMPIsDevice || LangOpts.SYCLIsDevice) &&
         PP.getAuxTargetInfo())
       InitializePredefinedMacros(*PP.getAuxTargetInfo(), LangOpts, FEOpts,
void clang::InitializePreprocessor(
   // Even with predefines off, some macros are still predefined.
   // These should all be defined in the preprocessor according to the
   // current language configuration.
   InitializeStandardPredefinedMacros(PP.getTargetInfo(), PP.getLangOpts(),
                                      FEOpts, Builder);
 
   // Add on the predefines from the driver.  Wrap in a #line directive to report
   // that they come from the command line.
-  if (!PP.getLangOpts().AsmPreprocessor)
-    Builder.append("# 1 \"<command line>\" 1");
 
   // Process #define's and #undef's in the order they are given.
   for (unsigned i = 0, e = InitOpts.Macros.size(); i != e; ++i) {
     if (InitOpts.Macros[i].second)  // isUndef
       Builder.undefineMacro(InitOpts.Macros[i].first);
     else
       DefineBuiltinMacro(Builder, InitOpts.Macros[i].first,
                          PP.getDiagnostics());
   }
 
   // Exit the command line and go back to <built-in> (2 is LC_LEAVE).
-  if (!PP.getLangOpts().AsmPreprocessor)
-    Builder.append("# 1 \"<built-in>\" 2");
 
   // If -imacros are specified, include them now.  These are processed before
   // any -include directives.
   for (unsigned i = 0, e = InitOpts.MacroIncludes.size(); i != e; ++i)
     AddImplicitIncludeMacros(Builder, InitOpts.MacroIncludes[i]);
 
   // Process -include-pch/-include-pth directives.
   if (!InitOpts.ImplicitPCHInclude.empty())

clang/lib/Lex/PPDirectives.cpp

case tok::eod:
     return;   // null directive.
   case tok::code_completion:
     setCodeCompletionReached();
     if (CodeComplete)
       CodeComplete->CodeCompleteDirective(
                                     CurPPLexer->getConditionalStackDepth() > 0);
     return;
   case tok::numeric_constant:  // # 7  GNU line marker directive.
-    if (getLangOpts().AsmPreprocessor)
-      break;  // # 4 is not a preprocessor directive in .S files.
+    // In a .S file "# 4" may be a comment so don't treat it as a preprocessor
+    // directive. However do permit it in the predefines file, as we use line
+    // markers to mark the builtin macros as being in a system header.
+    if (getLangOpts().AsmPreprocessor &&
+        SourceMgr.getFileID(SavedHash.getLocation()) != getPredefinesFileID())
+      break;
     return HandleDigitDirective(Result);
   default:
     IdentifierInfo *II = Result.getIdentifierInfo();
     if (!II) break; // Not an identifier.
 
     // Ask what the preprocessor keyword ID is.
     switch (II->getPPKeywordID()) {
     default: break;

clang/test/Preprocessor/directives_asm.S

+// RUN: %clang -c %s -o /dev/null 2>&1 | FileCheck %s
+
+// Check that preprocessor directives are recognised as such, but lines starting
+// with a # that aren't directives are instead treated as comments.
+
+#define MACRO .warning "This is a macro"
+        MACRO
+
+// CHECK: directives_asm.S:7:9: warning: This is a macro
+
+#not a preprocessing directive
+
+// CHECK-NOT: error: invalid preprocessing directive
+
+# 100
+
+        .warning "line number should not change"
+
+// CHECK: directives_asm.S:17:9: warning: line number should not change
+
+#line 100
+
+        .warning "line number should change"
+
+// CHECK: directives_asm.S:101:9: warning: line number should change

clang/test/Preprocessor/macro_redefined.S

+// RUN: %clang %s -E -DCLI_MACRO=1 2>&1 | FileCheck %s
+
+#define CLI_MACRO
+// CHECK: macro_redefined.S{{.+}}: warning: 'CLI_MACRO' macro redefined
+// CHECK: <command line>{{.+}}: note: previous definition is here
+
+#define REGULAR_MACRO
+#define REGULAR_MACRO 1
+// CHECK: macro_redefined.S{{.+}}: warning: 'REGULAR_MACRO' macro redefined
+// CHECK: macro_redefined.S{{.+}}: note: previous definition is here