[Clang] Record tokens in attribute arguments for user-defined C++/C2x attributes

[Qix-]

Summary

Currently, user-defined attributes (e.g. those registered via Clang plugins) don't hold much information or allow for much to be done with them, as their argument tokens are discarded entirely in almost all cases.

However, since they are quite flexible with their syntax (pretty much anything can go in there), it would be massively helpful if plugins could be handed the pre-processed and parsed tokens as a list to be consumed by third-party plugins and the like.

This diff creates a trailing data list for ParsedAttr objects and places the "recorded" tokens there from the lexer. I would have piggy-backed off of the normal backtrack system but unfortunately it doesn't call the handler for tokens being replayed from the backtrack cache, which in many cases we *do* care about. So I had to create a second recording callback.

The new plugin directory shows how this would be used from a plugin.

NOTE: I'm posting a bit prematurely, as on my WSL1 machine this caused a random test to fail, and I'm not entirely sure why. Hoping someone else can point me in the right direction to debug it.

The three tests that failed before hand were:

Failed Tests (3):
  Clang :: Frontend/plugin-delayed-template.cpp
  libc++ :: libcxx/gdb/gdb_pretty_printer_test.sh.cpp
  libc++ :: std/numerics/numeric.ops/numeric.ops.midpoint/midpoint.float.pass.cpp

And after:

Failed Tests (4):
  Clang :: Frontend/plugin-delayed-template.cpp
  Clang :: SemaObjC/externally-retained-no-arc.m
  libc++ :: libcxx/gdb/gdb_pretty_printer_test.sh.cpp
  libc++ :: std/numerics/numeric.ops/numeric.ops.midpoint/midpoint.float.pass.cpp

EDIT: Looks like the errors were transient on my end; successive runs of the identical changes on WSL1 produces different failures each time. CI appears to be green so hopefully everything is OK.

[Qix-]

Updated the diff to include a lot more context (-U9999). Thanks again for the tip :)

[aaron.ballman]

Thank you for this patch, I think it's really useful functionality for plugin authors!

Adding some additional reviewers for more opinions on the changes in the preprocessor.

clang/examples/PrintAttributeTokens/CMakeLists.txt

3–10

I'm not certain what this is actually doing -- the .exports file is empty (and no other plugin has this sort of .exports file thing). Is this needed?

clang/examples/PrintAttributeTokens/PrintAttributeTokens.cpp

37

Should probably use llvm::outs() instead (here and elsewhere in the plugin).

44–45

Per the usual coding conventions.

48–50

It'd probably be handy to also print identifier tokens.

clang/include/clang/Sema/ParsedAttr.h

236–238

Rather than use a bit-field for this, I'd add it as a regular unsigned after the declaration of UnavailableLoc.

284

486

I think it'd be reasonable to return nullptr in this case, WDYT?

clang/lib/Parse/ParseDeclCXX.cpp

4100–4102

I think plugins will expect these tokens to be available regardless of which attribute syntax is used, so you may need to look into also doing work for GNU and declspec attribute argument lists.

4121–4122

To be clear, we should have at least *two* tokens, right? One for the left paren and one for the right?

clang/test/Frontend/plugin-print-attr-tokens.cpp

9–12

Can you also verify that the token stream output from the plugin looks correct?

Can you also add additional tests for GNU and declspec attribute syntaxes?

[Qix-]

clang/examples/PrintAttributeTokens/CMakeLists.txt

3–10

Most likely not - the directory was copied from the PrintFunctionNames directory, which also has an empty exports file. I'm not completely familiar with the LLVM build configs so I figured it was required. Happy to remove it if need be :)

clang/examples/PrintAttributeTokens/PrintAttributeTokens.cpp

37

Sure thing, though should PrintFunctionNames be updated as well? Mostly copied the conventions over from that.

48–50

Yeah I caught that the other day too, thanks for reminding me.

clang/include/clang/Sema/ParsedAttr.h

284

Agreed; should 279 be updated too?

486

Sure, that's fine. I was trying to think why I did the assertion but nullptr seems fine too.

clang/lib/Parse/ParseDeclCXX.cpp

4100–4102

As far as I understand (and perhaps poorly communicated in the comment) is that plugin-defined attributes will always hit this codepath as opposed to the built-in attribute parsers.

I could be wrong here, though. Are arbitrary tokens even allowed in GNU/Declspec attributes? I thought it was just the C2x/C++ attributes syntax that allowed non-identifier tokens in the first place.

Either way, from what I could tell (trying a few different implementations of this change), this is the only place where user-defined attributes are parsed. I certainly could be missing something though.

4121–4122

This was confusing for me too at first.

By 4112, the left paren token has already been emitted by the lexer; ConsumeParen advances to the next token *after* the left-paren; _that_ token then becomes the first in the recording session.

Further, SkipUntil(tok::r_paren) checks if the current token is an r_paren, and if not, advances and then starts over. It doesn't advance _then_ check.

It's the presence of the left paren that advances us to this stage of the parsing in the first place and thus is emitted before our recording session starts. An empty argument list ([[some_attr()]]) thus only has an r_paren token emitted to the recording session, which is then ignored.

This is the existing design of the lexer it seems - I wasn't expecting the "reversed" order of operations, and the names of the lexer methods didn't help, either :)

clang/test/Frontend/plugin-print-attr-tokens.cpp

9–12

Can you also verify that the token stream output from the plugin looks correct?

Absolutely; how do I do this? The attributes don't make it into the final AST as far as I can tell, so is there a filecheck mechanism for checking the stdio streams directly? Is there an existing test that can do this I can reference?

Can you also add additional tests for GNU and declspec attribute syntaxes?

See comment above about GNU/declspec syntaxes. Depending on the outcome there, I'll certainly add them here.

[Qix-]

So I went back and checked and I remember why I didn't add explicit support for GNU/declspec attributes - they actually perform symbol lookups in the surrounding scope. I believe there's an issue right now with plugins that GNU-style attributes parser assumes that they're being parsed from the Tablegen stuff and thus expect the attribute classes to be parameterized as e.g. taking arguments or not. Therefore, putting arbitrary tokens in GNU-style and (as far as I can tell) declspec-style attributes is expressely not supported by the compiler. Also I believe this makes sense, too, because those syntaxes aren't spec'd to allow arbitrary tokens like C2x and C++ attributes are.

In fact, in my testing, the GNU-style attributes in plugins can't accept arguments whatsoever without getting some sort of compilation error. Maybe I'm not using the plugin system correctly, though.

[Qix-]

clang/lib/Parse/ParseDeclCXX.cpp

4100–4102

See my latest top-level comment.

[Qix-]

@aaron.ballman updated with comments.

[urnathan]

See also https://bugs.llvm.org/show_bug.cgi?id=46446. when I first fell into this issue, I did think it was trying to save the token stream as this patch is doing. Neat I thought :) although I'm a clang weenie, saving the tokens is putting this into deferred-parse territory, which makes me nervous. Wouldn't it be better for the ParsedAttrInfo objects to determine whether and how to parse their arguments. They could do so immediately, or save tokens, or whatever on a per-attribute per-argument basis. Isn't that more flexible? Add some ParsedAttrInfo defaults for the default cxx11, gnu & clang flavours of attributes?

[aaron.ballman]

In D99861#2694605, @urnathan wrote:

See also https://bugs.llvm.org/show_bug.cgi?id=46446. when I first fell into this issue, I did think it was trying to save the token stream as this patch is doing. Neat I thought :) although I'm a clang weenie, saving the tokens is putting this into deferred-parse territory, which makes me nervous.

FWIW, I've been stewing on the deferred parse issues with the approach from this patch because they make me nervous as well. Because [[]] attributes can have arbitrary token soup for their arguments and that includes arbitrary expressions, I worry that just giving the plugin author a bunch of tokens and saying "you deal with it" will only work for very trivial arguments.

Wouldn't it be better for the ParsedAttrInfo objects to determine whether and how to parse their arguments. They could do so immediately, or save tokens, or whatever on a per-attribute per-argument basis. Isn't that more flexible? Add some ParsedAttrInfo defaults for the default cxx11, gnu & clang flavours of attributes?

I think this would be an improved approach over just replaying the tokens for the plugin to handle. I would hope that we'd be able to design something that handles the easy cases of passing a string literal, an integer value, an enumerated list of named values, etc so that plugin authors don't have to reinvent the wheel for basic needs.

[Qix-]

I'm not sure exactly how to continue after the last few comments - what should the approach be for this patch? Or are these things we can shoot for in later patches?

[aaron.ballman]

In D99861#2697449, @Qix- wrote:

I'm not sure exactly how to continue after the last few comments - what should the approach be for this patch? Or are these things we can shoot for in later patches?

I don't think they're things we should shoot for in a later patch; the token replay approach doesn't seem like it would work for more complicated attribute arguments. As a concrete example of what would be super difficult to support would be:

struct S {
  int member;
  void func(int i, int j) [[plugin::attr(i + j + member)]];
};

because it would be very difficult to recognize that member is looked up in the context of the declaration of S::func() after having left the parsing context for the structure.

I think the one way to continue is along the lines of what's described in https://bugs.llvm.org/show_bug.cgi?id=46446#c11 so that the plugin handles parsing the arguments by invoking calls on the parser.

[Qix-]

Closing in favor of more complete plugin attribute changes discussed in IRC.

clang/examples/CMakeLists.txt

 if(NOT CLANG_BUILD_EXAMPLES)
   set_property(DIRECTORY PROPERTY EXCLUDE_FROM_ALL ON)
   set(EXCLUDE_FROM_ALL ON)
 endif()
 
 add_subdirectory(clang-interpreter)
 add_subdirectory(PrintFunctionNames)
+add_subdirectory(PrintAttributeTokens)
 add_subdirectory(AnnotateFunctions)
 add_subdirectory(Attribute)
 add_subdirectory(CallSuperAttribute)

clang/examples/PrintAttributeTokens/CMakeLists.txt

+# If we don't need RTTI or EH, there's no reason to export anything
+# from the plugin.
+if( NOT MSVC ) # MSVC mangles symbols differently, and
+               # PrintAttributeTokens.export contains C++ symbols.
+  if( NOT LLVM_REQUIRES_RTTI )
+    if( NOT LLVM_REQUIRES_EH )
+      set(LLVM_EXPORTED_SYMBOL_FILE ${CMAKE_CURRENT_SOURCE_DIR}/PrintAttributeTokens.exports)
+    endif()
+  endif()
+endif()
+
+add_llvm_library(PrintAttributeTokens MODULE PrintAttributeTokens.cpp PLUGIN_TOOL clang)
+
+if(LLVM_ENABLE_PLUGINS AND (WIN32 OR CYGWIN))
+  set(LLVM_LINK_COMPONENTS
+    Support
+  )
+  clang_target_link_libraries(PrintAttributeTokens PRIVATE
+    clangAST
+    clangBasic
+    clangFrontend
+    )
+endif()

clang/examples/PrintAttributeTokens/PrintAttributeTokens.cpp

+//===- PrintAttributeTokens.cpp
+//---------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Example clang plugin which prints all argument tokens from within a
+// [[print_tokens(...)]] attribute.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Lex/Token.h"
+#include "clang/Sema/ParsedAttr.h"
+#include "clang/Sema/Sema.h"
+#include "clang/Sema/SemaDiagnostic.h"
+#include "llvm/Support/raw_ostream.h"
+using namespace clang;
+
+namespace {
+
+struct PrintTokensAttrInfo : public ParsedAttrInfo {
+  PrintTokensAttrInfo() {
+    // GNU-style __attribute__(("example")) and C++-style [[example]] and
+    // [[plugin::example]] supported.
+    static constexpr Spelling S[] = {
+        {ParsedAttr::AS_GNU, "print_tokens"},
+        {ParsedAttr::AS_CXX11, "print_tokens"},
+        {ParsedAttr::AS_CXX11, "plugin::print_tokens"}};
+    Spellings = S;
+  }
+
+  AttrHandling handleDeclAttribute(Sema &S, Decl *D,
+                                   const ParsedAttr &Attr) const override {
+    llvm::errs() << "PrintAttributeTokens -----------------------\n";
+    D->dump(llvm::errs());
+    llvm::errs() << "\n";
+
+    if (Attr.getNumTokens() == 0) {
+      llvm::errs() << "<no tokens>\n";
+    } else {
+      const Token *tokens = Attr.getTokens();
+      for (unsigned i = 0; i < Attr.getNumTokens(); i++) {
+        llvm::errs() << tokens[i].getName();
+
+        if (tokens[i].isLiteral()) {
+          llvm::errs() << "\t=" << tokens[i].getLiteralData();
+        }
+
+        llvm::errs() << "\n";
+      }
+    }
+
+    llvm::errs() << "\n--------------------------------------------\n";
+
+    return AttributeApplied;
+  }
+};
+
+} // namespace
+
+static ParsedAttrInfoRegistry::Add<PrintTokensAttrInfo>
+    X("print_tokens", "print tokens inside the attribute arguments");

clang/examples/PrintAttributeTokens/README.txt

+This is a simple example demonstrating how to use clang's facility for
+registering custom attribute processors using a plugin.
+
+Build the plugin by running `make` in this directory.
+
+Once the plugin is built, you can run it using:
+--
+Linux:
+$ clang -cc1 -load ../../Debug+Asserts/lib/libPrintAttributeTokens.so -plugin print-attr-tokens some-input-file.c
+
+Mac:
+$ clang -cc1 -load ../../Debug+Asserts/lib/libPrintAttributeTokens.dylib -plugin print-attr-tokens some-input-file.c

clang/include/clang/Lex/Preprocessor.h

 /// Lexers know only about tokens within a single source file, and don't
 /// know anything about preprocessor-level issues like the \#include stack,
 /// token expansion, etc.
 class Preprocessor {
   friend class VAOptDefinitionContext;
   friend class VariadicMacroScopeGuard;
 
   llvm::unique_function<void(const clang::Token &)> OnToken;
+  llvm::unique_function<void(const clang::Token &)> OnRecordedToken;
   std::shared_ptr<PreprocessorOptions> PPOpts;
   DiagnosticsEngine        *Diags;
   LangOptions       &LangOpts;
   const TargetInfo *Target = nullptr;
   const TargetInfo *AuxTarget = nullptr;
   FileManager       &FileMgr;
   SourceManager     &SourceMgr;
   std::unique_ptr<ScratchBuffer> ScratchBuf;
void overrideMaxTokens(unsigned Value, SourceLocation Loc) {
     MaxTokensOverrideLoc = Loc;
   };
 
   SourceLocation getMaxTokensOverrideLoc() const { return MaxTokensOverrideLoc; }
 
   /// Register a function that would be called on each token in the final
   /// expanded token stream.
   /// This also reports annotation tokens produced by the parser.
+  ///
+  /// Note that the callback is not invoked for tokens re-emitted from the
+  /// backtrack cache. To receive callbacks for *any* tokens returned from
+  /// Lex(), use setTokenRecorder() instead.
   void setTokenWatcher(llvm::unique_function<void(const clang::Token &)> F) {
     OnToken = std::move(F);
   }
 
+  /// True if an active token watcher function has been registered with
+  /// setTokenWatcher().
+  bool hasTokenWatcher() const { return bool(OnToken); }
+
+  /// Register a function that would be called on each token returned by Lex().
+  /// This also reports annotation tokens produced by the parser.
+  ///
+  /// Note that the callback is invoked for tokens re-emitted from the backtrack
+  /// cache. Depending on how the Lexer is used between calls to
+  /// setTokenRecorder(), the callback may be called multiple times for the
+  /// same token. To avoid this and only receive fresh tokens from the
+  /// underlying lexer, use setTokenWatcher() instead.
+  void setTokenRecorder(llvm::unique_function<void(const clang::Token &)> F) {
+    OnRecordedToken = std::move(F);
+  }
+
+  /// True if an active token recorder function has been registered with
+  /// setTokenRecorder().
+  bool hasTokenRecorder() const { return bool(OnRecordedToken); }
+
   void setPreprocessToken(bool Preprocess) { PreprocessToken = Preprocess; }
 
   bool isMacroDefined(StringRef Id) {
     return isMacroDefined(&Identifiers.get(Id));
   }
   bool isMacroDefined(const IdentifierInfo *II) {
     return II->hasMacroDefinition() &&
            (!getLangOpts().Modules || (bool)getMacroDefinition(II));

clang/include/clang/Sema/ParsedAttr.h

 
 #ifndef LLVM_CLANG_SEMA_ATTRIBUTELIST_H
 #define LLVM_CLANG_SEMA_ATTRIBUTELIST_H
 
 #include "clang/Basic/AttrSubjectMatchRules.h"
 #include "clang/Basic/AttributeCommonInfo.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/SourceLocation.h"
+#include "clang/Lex/Token.h"
 #include "clang/Sema/Ownership.h"
 #include "llvm/ADT/PointerUnion.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/TinyPtrVector.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/Registry.h"
 #include "llvm/Support/VersionTuple.h"
 #include <cassert>

 ///
 /// 1: __attribute__(( const )). ParmName/Args/NumArgs will all be unused.
 /// 2: __attribute__(( mode(byte) )). ParmName used, Args/NumArgs unused.
 /// 3: __attribute__(( format(printf, 1, 2) )). ParmName/Args/NumArgs all used.
 /// 4: __attribute__(( aligned(16) )). ParmName is unused, Args/Num used.
 ///
 class ParsedAttr final
     : public AttributeCommonInfo,
-      private llvm::TrailingObjects<
-          ParsedAttr, ArgsUnion, detail::AvailabilityData,
-          detail::TypeTagForDatatypeData, ParsedType, detail::PropertyData> {
+      private llvm::TrailingObjects<ParsedAttr, ArgsUnion,
+                                    detail::AvailabilityData,
+                                    detail::TypeTagForDatatypeData, ParsedType,
+                                    detail::PropertyData, Token> {
   friend TrailingObjects;
 
   size_t numTrailingObjects(OverloadToken<ArgsUnion>) const { return NumArgs; }
+  size_t numTrailingObjects(OverloadToken<Token>) const { return NumTokens; }
   size_t numTrailingObjects(OverloadToken<detail::AvailabilityData>) const {
     return IsAvailability;
   }
   size_t
       numTrailingObjects(OverloadToken<detail::TypeTagForDatatypeData>) const {
     return IsTypeTagForDatatype;
   }
   size_t numTrailingObjects(OverloadToken<ParsedType>) const {
     return HasParsedType;
   }
   size_t numTrailingObjects(OverloadToken<detail::PropertyData>) const {
     return IsProperty;
   }
 
 private:
   IdentifierInfo *MacroII = nullptr;
   SourceLocation MacroExpansionLoc;
   SourceLocation EllipsisLoc;
 
   /// The number of expression arguments this attribute has.
   /// The expressions themselves are stored after the object.
   unsigned NumArgs : 16;
 
+  /// The number of tokens within the argument.
+  /// The tokens themselves are stored after the object.
+  unsigned NumTokens : 32;
+
   /// True if already diagnosed as invalid.
   mutable unsigned Invalid : 1;
 
   /// True if this attribute was used as a type attribute.
   mutable unsigned UsedAsTypeAttr : 1;
 
   /// True if this has the extra information associated with an
   /// availability attribute.
private:
 
   const ParsedAttrInfo &Info;
 
   ArgsUnion *getArgsBuffer() { return getTrailingObjects<ArgsUnion>(); }
   ArgsUnion const *getArgsBuffer() const {
     return getTrailingObjects<ArgsUnion>();
   }
 
+  Token *getTokensBuffer() { return getTrailingObjects<Token>(); }
+  Token const *getTokensBuffer() const { return getTrailingObjects<Token>(); }
+
   detail::AvailabilityData *getAvailabilityData() {
     return getTrailingObjects<detail::AvailabilityData>();
   }
   const detail::AvailabilityData *getAvailabilityData() const {
     return getTrailingObjects<detail::AvailabilityData>();
   }
 
 private:
   friend class AttributeFactory;
   friend class AttributePool;
 
   /// Constructor for attributes with expression arguments.
   ParsedAttr(IdentifierInfo *attrName, SourceRange attrRange,
              IdentifierInfo *scopeName, SourceLocation scopeLoc,
              ArgsUnion *args, unsigned numArgs, Syntax syntaxUsed,
-             SourceLocation ellipsisLoc)
+             Token *tokens = nullptr, unsigned numTokens = 0,
+             SourceLocation ellipsisLoc = SourceLocation())
       : AttributeCommonInfo(attrName, scopeName, attrRange, scopeLoc,
                             syntaxUsed),
-        EllipsisLoc(ellipsisLoc), NumArgs(numArgs), Invalid(false),
-        UsedAsTypeAttr(false), IsAvailability(false),
+        EllipsisLoc(ellipsisLoc), NumArgs(numArgs), NumTokens(numTokens),
+        Invalid(false), UsedAsTypeAttr(false), IsAvailability(false),
         IsTypeTagForDatatype(false), IsProperty(false), HasParsedType(false),
         HasProcessingCache(false), IsPragmaClangAttribute(false),
         Info(ParsedAttrInfo::get(*this)) {
     if (numArgs)
       memcpy(getArgsBuffer(), args, numArgs * sizeof(ArgsUnion));
+    if (numTokens)
+      memcpy(getTokensBuffer(), tokens, numTokens * sizeof(Token));
   }
 
   /// Constructor for availability attributes.
   ParsedAttr(IdentifierInfo *attrName, SourceRange attrRange,
              IdentifierInfo *scopeName, SourceLocation scopeLoc,
              IdentifierLoc *Parm, const AvailabilityChange &introduced,
              const AvailabilityChange &deprecated,
              const AvailabilityChange &obsoleted, SourceLocation unavailable,
              const Expr *messageExpr, Syntax syntaxUsed, SourceLocation strict,
              const Expr *replacementExpr)
       : AttributeCommonInfo(attrName, scopeName, attrRange, scopeLoc,
                             syntaxUsed),
-        NumArgs(1), Invalid(false), UsedAsTypeAttr(false), IsAvailability(true),
-        IsTypeTagForDatatype(false), IsProperty(false), HasParsedType(false),
-        HasProcessingCache(false), IsPragmaClangAttribute(false),
-        UnavailableLoc(unavailable), MessageExpr(messageExpr),
-        Info(ParsedAttrInfo::get(*this)) {
+        NumArgs(1), NumTokens(0), Invalid(false), UsedAsTypeAttr(false),
+        IsAvailability(true), IsTypeTagForDatatype(false), IsProperty(false),
+        HasParsedType(false), HasProcessingCache(false),
+        IsPragmaClangAttribute(false), UnavailableLoc(unavailable),
+        MessageExpr(messageExpr), Info(ParsedAttrInfo::get(*this)) {
     ArgsUnion PVal(Parm);
     memcpy(getArgsBuffer(), &PVal, sizeof(ArgsUnion));
     new (getAvailabilityData()) detail::AvailabilityData(
         introduced, deprecated, obsoleted, strict, replacementExpr);
   }
 
   /// Constructor for objc_bridge_related attributes.
   ParsedAttr(IdentifierInfo *attrName, SourceRange attrRange,
              IdentifierInfo *scopeName, SourceLocation scopeLoc,
              IdentifierLoc *Parm1, IdentifierLoc *Parm2, IdentifierLoc *Parm3,
              Syntax syntaxUsed)
       : AttributeCommonInfo(attrName, scopeName, attrRange, scopeLoc,
                             syntaxUsed),
-        NumArgs(3), Invalid(false), UsedAsTypeAttr(false),
+        NumArgs(3), NumTokens(0), Invalid(false), UsedAsTypeAttr(false),
         IsAvailability(false), IsTypeTagForDatatype(false), IsProperty(false),
         HasParsedType(false), HasProcessingCache(false),
         IsPragmaClangAttribute(false), Info(ParsedAttrInfo::get(*this)) {
     ArgsUnion *Args = getArgsBuffer();
     Args[0] = Parm1;
     Args[1] = Parm2;
     Args[2] = Parm3;
   }
 
   /// Constructor for type_tag_for_datatype attribute.
   ParsedAttr(IdentifierInfo *attrName, SourceRange attrRange,
              IdentifierInfo *scopeName, SourceLocation scopeLoc,
              IdentifierLoc *ArgKind, ParsedType matchingCType,
              bool layoutCompatible, bool mustBeNull, Syntax syntaxUsed)
       : AttributeCommonInfo(attrName, scopeName, attrRange, scopeLoc,
                             syntaxUsed),
-        NumArgs(1), Invalid(false), UsedAsTypeAttr(false),
+        NumArgs(1), NumTokens(0), Invalid(false), UsedAsTypeAttr(false),
         IsAvailability(false), IsTypeTagForDatatype(true), IsProperty(false),
         HasParsedType(false), HasProcessingCache(false),
         IsPragmaClangAttribute(false), Info(ParsedAttrInfo::get(*this)) {
     ArgsUnion PVal(ArgKind);
     memcpy(getArgsBuffer(), &PVal, sizeof(ArgsUnion));
     detail::TypeTagForDatatypeData &ExtraData = getTypeTagForDatatypeDataSlot();
     new (&ExtraData.MatchingCType) ParsedType(matchingCType);
     ExtraData.LayoutCompatible = layoutCompatible;
     ExtraData.MustBeNull = mustBeNull;
   }
 
   /// Constructor for attributes with a single type argument.
   ParsedAttr(IdentifierInfo *attrName, SourceRange attrRange,
              IdentifierInfo *scopeName, SourceLocation scopeLoc,
              ParsedType typeArg, Syntax syntaxUsed)
       : AttributeCommonInfo(attrName, scopeName, attrRange, scopeLoc,
                             syntaxUsed),
-        NumArgs(0), Invalid(false), UsedAsTypeAttr(false),
+        NumArgs(0), NumTokens(0), Invalid(false), UsedAsTypeAttr(false),
         IsAvailability(false), IsTypeTagForDatatype(false), IsProperty(false),
         HasParsedType(true), HasProcessingCache(false),
         IsPragmaClangAttribute(false), Info(ParsedAttrInfo::get(*this)) {
     new (&getTypeBuffer()) ParsedType(typeArg);
   }
 
   /// Constructor for microsoft __declspec(property) attribute.
   ParsedAttr(IdentifierInfo *attrName, SourceRange attrRange,
              IdentifierInfo *scopeName, SourceLocation scopeLoc,
              IdentifierInfo *getterId, IdentifierInfo *setterId,
              Syntax syntaxUsed)
       : AttributeCommonInfo(attrName, scopeName, attrRange, scopeLoc,
                             syntaxUsed),
-        NumArgs(0), Invalid(false), UsedAsTypeAttr(false),
+        NumArgs(0), NumTokens(0), Invalid(false), UsedAsTypeAttr(false),
         IsAvailability(false), IsTypeTagForDatatype(false), IsProperty(true),
         HasParsedType(false), HasProcessingCache(false),
         IsPragmaClangAttribute(false), Info(ParsedAttrInfo::get(*this)) {
     new (&getPropertyDataBuffer()) detail::PropertyData(getterId, setterId);
   }
 
   /// Type tag information is stored immediately following the arguments, if
   /// any, at the end of the object.  They are mutually exclusive with
public:
   void setIsPragmaClangAttribute() { IsPragmaClangAttribute = true; }
 
   bool isPackExpansion() const { return EllipsisLoc.isValid(); }
   SourceLocation getEllipsisLoc() const { return EllipsisLoc; }
 
   /// getNumArgs - Return the number of actual arguments to this attribute.
   unsigned getNumArgs() const { return NumArgs; }
 
+  /// getNumTokens - Return the number of tokens recorded by this attribute.
+  unsigned getNumTokens() const { return NumTokens; }
+
   /// getArg - Return the specified argument.
   ArgsUnion getArg(unsigned Arg) const {
     assert(Arg < NumArgs && "Arg access out of range!");
     return getArgsBuffer()[Arg];
   }
 
+  /// getTokens - Return the array of Tokens.
+  const Token *getTokens() const {
+    assert(NumTokens > 0 && "No Tokens to retrieve!");
+    return getTokensBuffer();
+  }
+
   bool isArgExpr(unsigned Arg) const {
     return Arg < NumArgs && getArg(Arg).is<Expr*>();
   }
 
   Expr *getArgAsExpr(unsigned Arg) const {
     return getArg(Arg).get<Expr*>();
   }
 

 /// A factory, from which one makes pools, from which one creates
 /// individual attributes which are deallocated with the pool.
 ///
 /// Note that it's tolerably cheap to create and destroy one of
 /// these as long as you don't actually allocate anything in it.
 class AttributeFactory {
 public:
   enum {
-    AvailabilityAllocSize =
-        ParsedAttr::totalSizeToAlloc<ArgsUnion, detail::AvailabilityData,
-                                     detail::TypeTagForDatatypeData, ParsedType,
-                                     detail::PropertyData>(1, 1, 0, 0, 0),
-    TypeTagForDatatypeAllocSize =
-        ParsedAttr::totalSizeToAlloc<ArgsUnion, detail::AvailabilityData,
-                                     detail::TypeTagForDatatypeData, ParsedType,
-                                     detail::PropertyData>(1, 0, 1, 0, 0),
-    PropertyAllocSize =
-        ParsedAttr::totalSizeToAlloc<ArgsUnion, detail::AvailabilityData,
-                                     detail::TypeTagForDatatypeData, ParsedType,
-                                     detail::PropertyData>(0, 0, 0, 0, 1),
+    AvailabilityAllocSize = ParsedAttr::totalSizeToAlloc<
+        ArgsUnion, detail::AvailabilityData, detail::TypeTagForDatatypeData,
+        ParsedType, detail::PropertyData, Token>(1, 1, 0, 0, 0, 0),
+    TypeTagForDatatypeAllocSize = ParsedAttr::totalSizeToAlloc<
+        ArgsUnion, detail::AvailabilityData, detail::TypeTagForDatatypeData,
+        ParsedType, detail::PropertyData, Token>(1, 0, 1, 0, 0, 0),
+    PropertyAllocSize = ParsedAttr::totalSizeToAlloc<
+        ArgsUnion, detail::AvailabilityData, detail::TypeTagForDatatypeData,
+        ParsedType, detail::PropertyData, Token>(0, 0, 0, 0, 1, 0),
   };
 
 private:
   enum {
     /// The number of free lists we want to be sure to support
     /// inline.  This is just enough that availability attributes
     /// don't surpass it.  It's actually very unlikely we'll see an
     /// attribute that needs more than that; on x86-64 you'd need 10
public:
   void takeAllFrom(AttributePool &pool) {
     takePool(pool);
     pool.Attrs.clear();
   }
 
   ParsedAttr *create(IdentifierInfo *attrName, SourceRange attrRange,
                      IdentifierInfo *scopeName, SourceLocation scopeLoc,
                      ArgsUnion *args, unsigned numArgs,
-                     ParsedAttr::Syntax syntax,
+                     ParsedAttr::Syntax syntax, Token *tokens = nullptr,
+                     unsigned numTokens = 0,
                      SourceLocation ellipsisLoc = SourceLocation()) {
-    size_t temp =
-        ParsedAttr::totalSizeToAlloc<ArgsUnion, detail::AvailabilityData,
-                                     detail::TypeTagForDatatypeData, ParsedType,
-                                     detail::PropertyData>(numArgs, 0, 0, 0, 0);
-    (void)temp;
     void *memory = allocate(
         ParsedAttr::totalSizeToAlloc<ArgsUnion, detail::AvailabilityData,
                                      detail::TypeTagForDatatypeData, ParsedType,
-                                     detail::PropertyData>(numArgs, 0, 0, 0,
-                                                           0));
-    return add(new (memory) ParsedAttr(attrName, attrRange, scopeName, scopeLoc,
-                                       args, numArgs, syntax, ellipsisLoc));
+                                     detail::PropertyData, Token>(
+            numArgs, 0, 0, 0, 0, numTokens));
+    return add(new (memory)
+                   ParsedAttr(attrName, attrRange, scopeName, scopeLoc, args,
+                              numArgs, syntax, tokens, numTokens, ellipsisLoc));
   }
 
   ParsedAttr *create(IdentifierInfo *attrName, SourceRange attrRange,
                      IdentifierInfo *scopeName, SourceLocation scopeLoc,
                      IdentifierLoc *Param, const AvailabilityChange &introduced,
                      const AvailabilityChange &deprecated,
                      const AvailabilityChange &obsoleted,
                      SourceLocation unavailable, const Expr *MessageExpr,
                      ParsedAttr::Syntax syntax, SourceLocation strict,
                      const Expr *ReplacementExpr) {
     void *memory = allocate(AttributeFactory::AvailabilityAllocSize);
     return add(new (memory) ParsedAttr(
         attrName, attrRange, scopeName, scopeLoc, Param, introduced, deprecated,
         obsoleted, unavailable, MessageExpr, syntax, strict, ReplacementExpr));
   }
 
   ParsedAttr *create(IdentifierInfo *attrName, SourceRange attrRange,
                      IdentifierInfo *scopeName, SourceLocation scopeLoc,
                      IdentifierLoc *Param1, IdentifierLoc *Param2,
                      IdentifierLoc *Param3, ParsedAttr::Syntax syntax) {
     void *memory = allocate(
         ParsedAttr::totalSizeToAlloc<ArgsUnion, detail::AvailabilityData,
                                      detail::TypeTagForDatatypeData, ParsedType,
-                                     detail::PropertyData>(3, 0, 0, 0, 0));
+                                     detail::PropertyData, Token>(3, 0, 0, 0, 0,
+                                                                  0));
     return add(new (memory) ParsedAttr(attrName, attrRange, scopeName, scopeLoc,
                                        Param1, Param2, Param3, syntax));
   }
 
   ParsedAttr *
   createTypeTagForDatatype(IdentifierInfo *attrName, SourceRange attrRange,
                            IdentifierInfo *scopeName, SourceLocation scopeLoc,
                            IdentifierLoc *argumentKind,
                            ParsedType matchingCType, bool layoutCompatible,
                            bool mustBeNull, ParsedAttr::Syntax syntax) {
     void *memory = allocate(AttributeFactory::TypeTagForDatatypeAllocSize);
     return add(new (memory) ParsedAttr(attrName, attrRange, scopeName, scopeLoc,
                                        argumentKind, matchingCType,
                                        layoutCompatible, mustBeNull, syntax));
   }
 
   ParsedAttr *createTypeAttribute(IdentifierInfo *attrName,
                                   SourceRange attrRange,
                                   IdentifierInfo *scopeName,
                                   SourceLocation scopeLoc, ParsedType typeArg,
                                   ParsedAttr::Syntax syntaxUsed) {
     void *memory = allocate(
         ParsedAttr::totalSizeToAlloc<ArgsUnion, detail::AvailabilityData,
                                      detail::TypeTagForDatatypeData, ParsedType,
-                                     detail::PropertyData>(0, 0, 0, 1, 0));
+                                     detail::PropertyData, Token>(0, 0, 0, 1, 0,
+                                                                  0));
     return add(new (memory) ParsedAttr(attrName, attrRange, scopeName, scopeLoc,
                                        typeArg, syntaxUsed));
   }
 
   ParsedAttr *
   createPropertyAttribute(IdentifierInfo *attrName, SourceRange attrRange,
                           IdentifierInfo *scopeName, SourceLocation scopeLoc,
                           IdentifierInfo *getterId, IdentifierInfo *setterId,
void clear() {
     clearListOnly();
     pool.clear();
   }
 
   /// Add attribute with expression arguments.
   ParsedAttr *addNew(IdentifierInfo *attrName, SourceRange attrRange,
                      IdentifierInfo *scopeName, SourceLocation scopeLoc,
                      ArgsUnion *args, unsigned numArgs,
-                     ParsedAttr::Syntax syntax,
+                     ParsedAttr::Syntax syntax, Token *tokens = nullptr,
+                     unsigned numTokens = 0,
                      SourceLocation ellipsisLoc = SourceLocation()) {
-    ParsedAttr *attr = pool.create(attrName, attrRange, scopeName, scopeLoc,
-                                   args, numArgs, syntax, ellipsisLoc);
+    ParsedAttr *attr =
+        pool.create(attrName, attrRange, scopeName, scopeLoc, args, numArgs,
+                    syntax, tokens, numTokens, ellipsisLoc);
     addAtEnd(attr);
     return attr;
   }
 
   /// Add availability attribute.
   ParsedAttr *addNew(IdentifierInfo *attrName, SourceRange attrRange,
                      IdentifierInfo *scopeName, SourceLocation scopeLoc,
                      IdentifierLoc *Param, const AvailabilityChange &introduced,

clang/lib/Lex/Preprocessor.cpp

void Preprocessor::Lex(Token &Result) {
 
   if ((LexLevel == 0 || PreprocessToken) &&
       !Result.getFlag(Token::IsReinjected)) {
     if (LexLevel == 0)
       ++TokenCount;
     if (OnToken)
       OnToken(Result);
   }
+
+  if (OnRecordedToken)
+    OnRecordedToken(Result);
 }
 
 /// Lex a header-name token (including one formed from header-name-tokens if
 /// \p AllowConcatenation is \c true).
 ///
 /// \param FilenameTok Filled in with the next token. On success, this will
 ///        be either a header_name token. On failure, it will be whatever other
 ///        token was found instead.

clang/lib/Parse/ParseDecl.cpp

void Parser::ParseAlignmentSpecifier(ParsedAttributes &Attrs,
 
   T.consumeClose();
   if (EndLoc)
     *EndLoc = T.getCloseLocation();
 
   ArgsVector ArgExprs;
   ArgExprs.push_back(ArgExpr.get());
   Attrs.addNew(KWName, KWLoc, nullptr, KWLoc, ArgExprs.data(), 1,
-               ParsedAttr::AS_Keyword, EllipsisLoc);
+               ParsedAttr::AS_Keyword, nullptr, 0, EllipsisLoc);
 }
 
 ExprResult Parser::ParseExtIntegerArgument() {
   assert(Tok.is(tok::kw__ExtInt) && "Not an extended int type");
   ConsumeToken();
 
   BalancedDelimiterTracker T(*this, tok::l_paren);
   if (T.expectAndConsume())

clang/lib/Parse/ParseDeclCXX.cpp

bool Parser::ParseCXX11AttributeArgs(IdentifierInfo *AttrName,
                                      SourceLocation ScopeLoc) {
   assert(Tok.is(tok::l_paren) && "Not a C++11 attribute argument list");
   SourceLocation LParenLoc = Tok.getLocation();
   const LangOptions &LO = getLangOpts();
   ParsedAttr::Syntax Syntax =
       LO.CPlusPlus ? ParsedAttr::AS_CXX11 : ParsedAttr::AS_C2x;
 
   // If the attribute isn't known, we will not attempt to parse any
-  // arguments.
+  // arguments. Instead, we just record the tokens and add the attribute
+  // directly. The recording happens here because this is the only place
+  // where user-defined (via plugins) attributes are parsed, and thus
+  // they care about the token stream directly.
   if (!hasAttribute(LO.CPlusPlus ? AttrSyntax::CXX : AttrSyntax::C, ScopeName,
                     AttrName, getTargetInfo(), getLangOpts())) {
-    // Eat the left paren, then skip to the ending right paren.
+    // Begin recording session.
+    SmallVector<Token> RecordedTokens;
+    assert(!PP.hasTokenRecorder());
+    PP.setTokenRecorder(
+        [&RecordedTokens](const Token &Tok) { RecordedTokens.push_back(Tok); });
+
+    // Eat the left paren.
     ConsumeParen();
+
+    // skip to the ending right paren.
     SkipUntil(tok::r_paren);
-    return false;
+
+    // End recording session.
+    PP.setTokenRecorder(nullptr);
+
+    // Add new attribute with the token list.
+    // We assert that we have at least one token,
+    // since we have to ignore the final r_paren.
+    assert(RecordedTokens.size() > 0);
+    Attrs.addNew(
+        AttrName,
+        SourceRange(ScopeLoc.isValid() ? ScopeLoc : AttrNameLoc, AttrNameLoc),
+        ScopeName, ScopeLoc, nullptr, 0,
+        getLangOpts().CPlusPlus ? ParsedAttr::AS_CXX11 : ParsedAttr::AS_C2x,
+        RecordedTokens.data(), RecordedTokens.size() - 2);
+
+    return true;
   }
 
   if (ScopeName && (ScopeName->isStr("gnu") || ScopeName->isStr("__gnu__"))) {
     // GNU-scoped attributes have some special cases to handle GNU-specific
     // behaviors.
     ParseGNUAttributeArgs(AttrName, AttrNameLoc, Attrs, EndLoc, ScopeName,
                           ScopeLoc, Syntax, nullptr);
     return true;

clang/lib/Sema/ParsedAttr.cpp

size_t ParsedAttr::allocated_size() const {
   if (IsAvailability) return AttributeFactory::AvailabilityAllocSize;
   else if (IsTypeTagForDatatype)
     return AttributeFactory::TypeTagForDatatypeAllocSize;
   else if (IsProperty)
     return AttributeFactory::PropertyAllocSize;
   else if (HasParsedType)
     return totalSizeToAlloc<ArgsUnion, detail::AvailabilityData,
                             detail::TypeTagForDatatypeData, ParsedType,
-                            detail::PropertyData>(0, 0, 0, 1, 0);
+                            detail::PropertyData, Token>(0, 0, 0, 1, 0, 0);
   return totalSizeToAlloc<ArgsUnion, detail::AvailabilityData,
                           detail::TypeTagForDatatypeData, ParsedType,
-                          detail::PropertyData>(NumArgs, 0, 0, 0, 0);
+                          detail::PropertyData, Token>(NumArgs, 0, 0, 0, 0,
+                                                       NumTokens);
 }
 
 AttributeFactory::AttributeFactory() {
   // Go ahead and configure all the inline capacity.  This is just a memset.
   FreeLists.resize(InlineFreeListsCapacity);
 }
 AttributeFactory::~AttributeFactory() = default;
 

clang/test/Frontend/plugin-print-attr-tokens.cpp

+// RUN: %clang -cc1 -load %llvmshlibdir/PrintAttributeTokens%pluginext -fsyntax-only -ast-dump -verify %s
+// REQUIRES: plugins, examples
+
+// expected-no-diagnostics
+[[print_tokens(
+    the, mitochondria,
+    <is + the + powerhouse>, Of::The($cell))]] void
+fn1a() {}
+[[plugin::print_tokens("a string")]] void fn1b() {}
+[[plugin::print_tokens()]] void fn1c() {}
+[[plugin::print_tokens(some_ident)]] void fn1d() {}
+[[plugin::print_tokens(int)]] void fn1e() {}