[Clang] Fix how ReadMacroParameterList handles comments in macro parameter-list when in -CC mode

[shafik]

Summary

Currently when in -CC mode when processing a function like macro ReadMacroParameterList(...) does not handle the case where there is a comment embedded in the parameter-list.

Instead of using LexUnexpandedToken(...) I switched to using LexUnexpandedNonComment(...) which eats comments while lexing.

Fixes: https://github.com/llvm/llvm-project/issues/60887

[erichkeane]

Based on the error message we had, it seems like this previous behavior was intentional. Are we sure comments SHOULD be allowed here? @tahonermann / @cor3ntin are the LEX code owners.

[cor3ntin]

This looks good to me.
Comments are allowed anywhere between token, anywhere whitespace would be.

[carsonRadtke]

It appears LexUnexpandedToken gets used multiple times in this function.

The following example demonstrates an error with each call not being LexUnexpandedNonComment:

/* no errors with comments removed */
#define A(.../**/)
#define B(x/**/)
#define C(x.../**/)

https://godbolt.org/z/Y6fGb59qb

Offending code is on lines 2663, 2697, and 2713, respectively.

[shafik]

• Switched to LexUnexpandedNonComment in more places
• Extended test suite to cover the new cases uncovered in the comments

[cor3ntin]

Sorry for the late review.
Can you add a release note mentioning the fixed issue?
Otherwise LGTM

[shafik]

• Add release notes.

[cor3ntin]

Thanks!

clang/docs/ReleaseNotes.rst

- Fix false-positive diagnostic issued for consteval initializers of temporary
   objects.
   (`#60286 <https://github.com/llvm/llvm-project/issues/60286>`_)
 - Correct restriction of trailing requirements clauses on a templated function.
   Previously we only rejected non-'templated' things, but the restrictions ALSO need
   to limit non-defined/non-member functions as well. Additionally, we now diagnose
   requires on lambdas when not allowed, which we previously missed.
   (`#61748 <https://github.com/llvm/llvm-project/issues/61748>`_)
 - Fix confusing diagnostic for incorrect use of qualified concepts names.
+- Fix handling of comments in function like macros so they are ignored in -CC
+  mode.
+  (`#60887 <https://github.com/llvm/llvm-project/issues/60887>`_)
 
 
 Bug Fixes to Compiler Builtins
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Bug Fixes to Attribute Support
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

clang/lib/Lex/PPDirectives.cpp

 /// ReadMacroParameterList - The ( starting a parameter list of a macro
 /// definition has just been read.  Lex the rest of the parameters and the
 /// closing ), updating MI with what we learn.  Return true if an error occurs
 /// parsing the param list.
 bool Preprocessor::ReadMacroParameterList(MacroInfo *MI, Token &Tok) {
   SmallVector<IdentifierInfo*, 32> Parameters;
 
   while (true) {
-    LexUnexpandedToken(Tok);
+    LexUnexpandedNonComment(Tok);
     switch (Tok.getKind()) {
     case tok::r_paren:
       // Found the end of the parameter list.
       if (Parameters.empty())  // #define FOO()
         return false;
       // Otherwise we have #define FOO(A,)
       Diag(Tok, diag::err_pp_expected_ident_in_arg_list);
       return true;
     case tok::ellipsis:  // #define X(... -> C99 varargs
       if (!LangOpts.C99)
         Diag(Tok, LangOpts.CPlusPlus11 ?
              diag::warn_cxx98_compat_variadic_macro :
              diag::ext_variadic_macro);
 
       // OpenCL v1.2 s6.9.e: variadic macros are not supported.
       if (LangOpts.OpenCL && !LangOpts.OpenCLCPlusPlus) {
         Diag(Tok, diag::ext_pp_opencl_variadic_macros);
       }
 
       // Lex the token after the identifier.
-      LexUnexpandedToken(Tok);
+      LexUnexpandedNonComment(Tok);
       if (Tok.isNot(tok::r_paren)) {
         Diag(Tok, diag::err_pp_missing_rparen_in_macro_def);
         return true;
       }
       // Add the __VA_ARGS__ identifier as a parameter.
       Parameters.push_back(Ident__VA_ARGS__);
       MI->setIsC99Varargs();
       MI->setParameterList(Parameters, BP);
default:
         Diag(Tok, diag::err_pp_duplicate_name_in_arg_list) << II;
         return true;
       }
 
       // Add the parameter to the macro info.
       Parameters.push_back(II);
 
       // Lex the token after the identifier.
-      LexUnexpandedToken(Tok);
+      LexUnexpandedNonComment(Tok);
 
       switch (Tok.getKind()) {
       default:          // #define X(A B
         Diag(Tok, diag::err_pp_expected_comma_in_arg_list);
         return true;
       case tok::r_paren: // #define X(A)
         MI->setParameterList(Parameters, BP);
         return false;
       case tok::comma:  // #define X(A,
         break;
       case tok::ellipsis:  // #define X(A... -> GCC extension
         // Diagnose extension.
         Diag(Tok, diag::ext_named_variadic_macro);
 
         // Lex the token after the identifier.
-        LexUnexpandedToken(Tok);
+        LexUnexpandedNonComment(Tok);
         if (Tok.isNot(tok::r_paren)) {
           Diag(Tok, diag::err_pp_missing_rparen_in_macro_def);
           return true;
         }
 
         MI->setIsGNUVarargs();
         MI->setParameterList(Parameters, BP);
         return false;

clang/test/Preprocessor/comment_save_macro.c

 // RUN: %clang_cc1 -E -C %s | FileCheck -check-prefix=CHECK-C -strict-whitespace %s
 // CHECK-C: boo bork bar // zot
+// CHECK-C: ( 0 );
+// CHECK-C: ( 0,1,2 );
+// CHECK-C: ( 0,1,2 );
 
 // RUN: %clang_cc1 -E -CC %s | FileCheck -check-prefix=CHECK-CC -strict-whitespace %s
 // CHECK-CC: boo bork /* blah*/ bar // zot
+// CHECK-CC: (/**/0/**/);
+// CHECK-CC: (/**/0,1,2/**/);
+// CHECK-CC: (/**/0,1,2/**/);
 
 // RUN: %clang_cc1 -E %s | FileCheck -strict-whitespace %s
 // CHECK: boo bork bar
+// CHECK: ( 0 );
+// CHECK: ( 0,1,2 );
+// CHECK: ( 0,1,2 );
 
 
 #define FOO bork // blah
 boo FOO bar // zot
+#define M(/**/x/**/) (/**/x/**/)
+M(0);
+#define M2(/**/.../**/) (/**/__VA_ARGS__/**/)
+M2(0,1,2);
+#define M3(/**/x.../**/) (/**/x/**/)
+M3(0,1,2);