[clang] extend external_source_symbol attribute with the USR clause

[arphaman]

Summary

Allow the user to specify a concrete USR in the external_source_symbol attribute. That will let Clang's indexer to use Swift USRs for Swift declarations that are represented with C++ declarations.

To allow the sources to conditionally enable this new extension, __has_feature is extended in Clang:

__has_feature(attribute_external_source_symbol_with_usr)

[erichkeane]

I'm disturbed that the string-literal diagnostic you changed never shows up in the tests. I suspect this attribute needs significantly better test coverage.

clang/docs/LanguageExtensions.rst

4815 ↗

(On Diff #487546)

Can you explain in here what the "USR" is here/means here? Or what said string is supposed to be?

4823 ↗

(On Diff #487546)

__has_feature isn't really the right solution here. Though, we don't typically have a way to test for individual features of an attribute other than the value that __has_attribute returns. In standard attributes, it returns a larger number when new features are added (see SD-10's Feature Test Macros).

I don't think I want us using __has_feature here though, either a different value for __has_attribute might be OK, or alternatively, perhaps just a simple feature-test-macro.

clang/include/clang/Basic/AttrDocs.td

1754

This DEFINITELY needs to better explain what a USR is, and why one would use it. Many times this document is used by folks to discover attributes they didn't know existed, and this needs to be usable for someone who doesn't really get the context until reading this document.

clang/include/clang/Basic/DiagnosticCommonKinds.td

58

I don't think adding 'name' to these makes them particularly more readable? At least from what I can see reading it here.

[arphaman]

Updated to address review comments.

[arphaman]

In D141324#4039629, @erichkeane wrote:

I'm disturbed that the string-literal diagnostic you changed never shows up in the tests. I suspect this attribute needs significantly better test coverage.

Please elaborate. All the diagnostics have appropriate coverage already.

clang/include/clang/Basic/DiagnosticCommonKinds.td

58

I did not add 'name' here, I moved it from the right hand side as I added a new clause that didn't use name.

[erichkeane]

In D141324#4041159, @arphaman wrote:

In D141324#4039629, @erichkeane wrote:

I'm disturbed that the string-literal diagnostic you changed never shows up in the tests. I suspect this attribute needs significantly better test coverage.

Please elaborate. All the diagnostics have appropriate coverage already.

I'd mistakenly thought that you'd changed the message to add 'name', not moved it.

That said, test coverage of the new feature should be improved. I don't see anything about the contents of the USR, nor anything with the USR field being dependent.

clang/include/clang/Basic/DiagnosticCommonKinds.td

58

Ah! Thanks! I missed that.

[bnbarham]

LGTM, thanks Alex!

[erichkeane]

Note both the Clang maintainer, and attributes maintainer have requested changes in some way, please let that finish before committing.

[arphaman]

I posted the Swift patch here:
https://github.com/apple/swift/pull/63002

[arphaman]

add more test coverage

[arphaman]

@erichkeane I've added more test coverage with different USR values, C++ decls and some dependent C++ decls too. How does it look now?

[erichkeane]

clang/include/clang/Basic/AttrDocs.td

1755

1762

1764

I don't think we can do this. __has_attribute needs to take the actual name of the attribute.

[aaron.ballman]

clang/include/clang/Basic/AttrDocs.td

1764

+1

clang/lib/Basic/Attributes.cpp

30–32 ↗

(On Diff #490214)

This is handled automagically for you in Attr.td, but under the correct attribute name instead of an invented name.

[arphaman]

clang/include/clang/Basic/AttrDocs.td

1764

__has_attribute(external_source_symbol) already works but it wouldn't help here as it doesn't tell whether the attribute supports USR or not. Are you saying I should come up with a totally new attribute that accepts USR? That was originally why I had the __has_feature check.

[erichkeane]

clang/include/clang/Basic/AttrDocs.td

1764

The typical way of doing this in the standard is to have has_c_attribute/has_cpp_attribute return a larger non-zero value for the normal attribute.

I suspect we should extend __has_attribute to do something similar (that is, allowing Feature Test Macros' for it). Aaron, WDYT?

[aaron.ballman]

clang/include/clang/Basic/AttrDocs.td

1764

The typical way of doing this in the standard is to have has_c_attribute/has_cpp_attribute return a larger non-zero value for the normal attribute.

Which we expose for the CXX11 and C2X spellings in Attr.td, but don't expose for the Clang spelling (easy enough to change).

I suspect we should extend __has_attribute to do something similar (that is, allowing Feature Test Macros' for it). Aaron, WDYT?

+1, if we don't already do that. This should be handled entirely within Attr.td and ClangAttrEmitter.cpp for a generic solution.

[arphaman]

clang/include/clang/Basic/AttrDocs.td

1764

sounds good, I will look into that and will update this patch.

[arphaman]

Updated __has_attribute(external_source_symbol) to return a version.

[erichkeane]

clang/include/clang/Basic/Attr.td

962

For standards version numbers, we tend to set this to a 'date' more or less, so something like 20230119. I wonder if there is value to making THAT how we do this here too?

clang/utils/TableGen/ClangAttrEmitter.cpp

3327

Why is this =="Clang" specific? Since you've added the Version to the spelling, I'd anticipate us to just be able to grab it for the current spelling. I wouldn't want an individual spelling here to override it, particularly since with this change Clang could potentially override the standards version.

[arphaman]

clang/include/clang/Basic/Attr.td

962

That's a good idea, I can update it to be a specific date.

clang/utils/TableGen/ClangAttrEmitter.cpp

3327

I needed it since there's no specific "Clang" variety that's being called for this function. Otherwise the "GNU" variety passed to the function doesn't match "Clang" variety in the record. What's the best way to compute the current spelling in this case?

[erichkeane]

clang/utils/TableGen/ClangAttrEmitter.cpp

3327

Hmm... that is strange. I would have expected this to be called for each of the individual spellings, it doesn't really make sense that it wouldn't pick it up from the base 'Spellings'. I've unfortunately not debugged this code generation in a while. BUT, we care about the 'version' on a per-spelling basis, so it would presumably need to 'pick' an actual spelling.

[aaron.ballman]

clang/utils/TableGen/ClangAttrEmitter.cpp

3327

This does get called for each of the base spellings: https://github.com/llvm/llvm-project/blob/main/clang/utils/TableGen/ClangAttrEmitter.cpp#L3393

I think the issue is that, in Attr.td, the GNU spelling has no version information associated with it. I think what should happen is that spelling should get a Version field the same as CXX11 and C2X do, and the Clang spelling can pass that information along to the GNU spelling so that it gets picked up here during tablegen.

One thing we should probably be careful of is compatibility with GCC. I *think* GCC just returns 0 or 1 from __has_attribute but we should find out if they return specific values for any of the attributes we already support (a follow-up patch can correct those values so we match GCC if necessary).

[arphaman]

clang/utils/TableGen/ClangAttrEmitter.cpp

3327

I think the issue is that, in Attr.td, the GNU spelling has no version information associated with it. I think what should happen is that spelling should get a Version field the same as CXX11 and C2X do, and the Clang spelling can pass that information along to the GNU spelling so that it gets picked up here during tablegen.

I'm not sure I understand how Clang should pass that information to the GNU spelling. Here we already set Version in each spelling. Do you think "Clang" should inherit from "GNU" spelling in tablegen, e.g.:

class Clang<string name, bit allowInC = 1, int version = 1>
    : GNU<name, "Clang", version> {
  bit AllowInC = allowInC;
}

[arphaman]

updated patch.

[arphaman]

clang/utils/TableGen/ClangAttrEmitter.cpp

3327

@aaron.ballman @erichkeane I updated this function to use FlattenedSpelling - now I don't have to check for "Clang" explicitly. How does it look now.

[arphaman]

@erichkeane ping.

[erichkeane]

That part is Ok now, thanks. However, this still needs a release note, plus a test for spelling versions.

[arphaman]

• Updated version to be the date.
• Updated release notes.
• Fixed a bug in my changed tablegen code in previous patch.

[arphaman]

updated test file with correct version check.

clang/docs/ReleaseNotes.rst

 --------------------------
 - Introduced a new function attribute ``__attribute__((unsafe_buffer_usage))``
   to be worn by functions containing buffer operations that could cause out of
   bounds memory accesses. It emits warnings at call sites to such functions when
   the flag ``-Wunsafe-buffer-usage`` is enabled.
 - ``__declspec`` attributes can now be used together with the using keyword. Before
   the attributes on ``__declspec`` was ignored, while now it will be forwarded to the
   point where the alias is used.
+- Introduced a new ``USR`` (unified symbol resolution) clause inside of the
+existing ``__attribute__((external_source_symbol))`` attribute. Clang's indexer
+uses the optional USR value when indexing Clang's AST. This value is expected
+to be generated by an external compiler when generating C++ bindings during
+the compilation of the foreign language sources (e.g. Swift).
 
 Improvements to Clang's diagnostics
 -----------------------------------
 - We now generate a diagnostic for signed integer overflow due to unary minus
   in a non-constant expression context.
   (`#31643 <https://github.com/llvm/llvm-project/issues/31643>`)
 - Clang now warns by default for C++20 and later about deprecated capture of
   ``this`` with a capture default of ``=``. This warning can be disabled with

clang/include/clang/Basic/Attr.td

 class VariadicEnumArgument<string name, string type, list<string> values,
                            list<string> enums> : Argument<name, 1>  {
   string Type = type;
   list<string> Values = values;
   list<string> Enums = enums;
 }
 
 // This handles one spelling of an attribute.
-class Spelling<string name, string variety> {
+class Spelling<string name, string variety, int version = 1> {
   string Name = name;
   string Variety = variety;
+  int Version = version;
 }
 
 class GNU<string name> : Spelling<name, "GNU">;
 class Declspec<string name> : Spelling<name, "Declspec">;
 class Microsoft<string name> : Spelling<name, "Microsoft">;
 class CXX11<string namespace, string name, int version = 1>
-    : Spelling<name, "CXX11"> {
+    : Spelling<name, "CXX11", version> {
   string Namespace = namespace;
-  int Version = version;
 }
 class C2x<string namespace, string name, int version = 1>
-    : Spelling<name, "C2x"> {
+    : Spelling<name, "C2x", version> {
   string Namespace = namespace;
-  int Version = version;
 }
 
 class Keyword<string name> : Spelling<name, "Keyword">;
 class Pragma<string namespace, string name> : Spelling<name, "Pragma"> {
   string Namespace = namespace;
 }
 
 // The GCC spelling implies GNU<name>, CXX11<"gnu", name>, and optionally,
 // C2x<"gnu", name>. This spelling should be used for any GCC-compatible
 // attributes.
 class GCC<string name, bit allowInC = 1> : Spelling<name, "GCC"> {
   bit AllowInC = allowInC;
 }
 
 // The Clang spelling implies GNU<name>, CXX11<"clang", name>, and optionally,
 // C2x<"clang", name>. This spelling should be used for any Clang-specific
 // attributes.
-class Clang<string name, bit allowInC = 1> : Spelling<name, "Clang"> {
+class Clang<string name, bit allowInC = 1, int version = 1>
+    : Spelling<name, "Clang", version> {
   bit AllowInC = allowInC;
 }
 
 // HLSL Semantic spellings
 class HLSLSemantic<string name> : Spelling<name, "HLSLSemantic">;
 
 class Accessor<string name, list<Spelling> spellings> {
   string Name = name;

 } }];
   let HasCustomParsing = 1;
   let InheritEvenIfAlreadyPresent = 1;
   let Subjects = SubjectList<[Named]>;
   let Documentation = [AvailabilityDocs];
 }
 
 def ExternalSourceSymbol : InheritableAttr {
-  let Spellings = [Clang<"external_source_symbol">];
+  let Spellings = [Clang<"external_source_symbol", /*allowInC=*/1,
+                   /*version=*/20230206>];
   let Args = [StringArgument<"language", 1>,
               StringArgument<"definedIn", 1>,
-              BoolArgument<"generatedDeclaration", 1>];
+              BoolArgument<"generatedDeclaration", 1>,
+              StringArgument<"USR", 1>];
   let HasCustomParsing = 1;
   let Subjects = SubjectList<[Named]>;
   let Documentation = [ExternalSourceSymbolDocs];
 }
 
 def Blocks : InheritableAttr {
   let Spellings = [Clang<"blocks">];
   let Args = [EnumArgument<"Type", "BlockType", ["byref"], ["ByRef"]>];

clang/include/clang/Basic/AttrDocs.td

language=\ *string-literal*
   The name of the source language in which this declaration was defined.
 
 defined_in=\ *string-literal*
   The name of the source container in which the declaration was defined. The
   exact definition of source container is language-specific, e.g. Swift's
   source containers are modules, so ``defined_in`` should specify the Swift
   module name.
 
+USR=\ *string-literal*
+  String that specifies a unified symbol resolution (USR) value for this
+  declaration. USR string uniquely identifies this particular declaration, and
+  is typically used when constructing an index of a codebase.
+  The USR value in this attribute is expected to be generated by an external
+  compiler that compiled the native declaration using its original source
+  language. The exact format of the USR string and its other attributes
+  are determined by the specification of this declaration's source language.
+  When not specified, Clang's indexer will use the Clang USR for this symbol.
+  User can query to see if Clang supports the use of the ``USR`` clause in
+  the ``external_source_symbol`` attribute with
+  ``__has_attribute(external_source_symbol) >= 20230206``.
+
 generated_declaration
   This declaration was automatically generated by some tool.
 
 The clauses can be specified in any order. The clauses that are listed above are
 all optional, but the attribute has to have at least one clause.
   }];
 }
 

clang/include/clang/Basic/DiagnosticCommonKinds.td

 let CategoryName = "Lexical or Preprocessor Issue" in {
 
 def err_expected_colon_after_setter_name : Error<
   "method name referenced in property setter attribute "
   "must end with ':'">;
 def err_expected_string_literal : Error<"expected string literal "
   "%select{in %1|for diagnostic message in static_assert|"
           "for optional message in 'availability' attribute|"
-          "for %select{language|source container}1 name in "
+          "for %select{language name|source container name|USR}1 in "
           "'external_source_symbol' attribute}0">;
 def err_invalid_string_udl : Error<
   "string literal with user-defined suffix cannot be used here">;
 def err_invalid_character_udl : Error<
   "character literal with user-defined suffix cannot be used here">;
 def err_invalid_numeric_udl : Error<
   "numeric literal with user-defined suffix cannot be used here">;
 

clang/include/clang/Basic/DiagnosticParseKinds.td

def err_availability_query_wildcard_required: Error<
   "must handle potential future platforms with '*'">;
 def err_availability_query_repeated_platform: Error<
   "version for '%0' already specified">;
 def err_availability_query_repeated_star : Error<
   "'*' query has already been specified">;
 
 // External source symbol attribute
 def err_external_source_symbol_expected_keyword : Error<
-  "expected 'language', 'defined_in', or 'generated_declaration'">;
+  "expected 'language', 'defined_in', 'generated_declaration', or 'USR'">;
 def err_external_source_symbol_duplicate_clause : Error<
   "duplicate %0 clause in an 'external_source_symbol' attribute">;
 
 // Type safety attributes
 def err_type_safety_unknown_flag : Error<
   "invalid comparison flag %0; use 'layout_compatible' or 'must_be_null'">;
 
 // Type traits

clang/include/clang/Parse/Parser.h

class Parser : public CodeCompletionHandler {
   /// Identifier for "strict".
   IdentifierInfo *Ident_strict;
 
   /// Identifier for "replacement".
   IdentifierInfo *Ident_replacement;
 
   /// Identifiers used by the 'external_source_symbol' attribute.
   IdentifierInfo *Ident_language, *Ident_defined_in,
-      *Ident_generated_declaration;
+      *Ident_generated_declaration, *Ident_USR;
 
   /// C++11 contextual keywords.
   mutable IdentifierInfo *Ident_final;
   mutable IdentifierInfo *Ident_GNU_final;
   mutable IdentifierInfo *Ident_override;
 
   // C++2a contextual keywords.
   mutable IdentifierInfo *Ident_import;

clang/lib/Index/USRGeneration.cpp

 bool clang::index::generateUSRForDecl(const Decl *D,
                                       SmallVectorImpl<char> &Buf) {
   if (!D)
     return true;
   // We don't ignore decls with invalid source locations. Implicit decls, like
   // C++'s operator new function, can have invalid locations but it is fine to
   // create USRs that can identify them.
 
+  // Check if the declaration has explicit external USR specified.
+  auto *CD = D->getCanonicalDecl();
+  if (auto *ExternalSymAttr = CD->getAttr<ExternalSourceSymbolAttr>()) {
+    if (!ExternalSymAttr->getUSR().empty()) {
+      llvm::raw_svector_ostream Out(Buf);
+      Out << ExternalSymAttr->getUSR();
+      return false;
+    }
+  }
   USRGenerator UG(&D->getASTContext(), Buf);
   UG.Visit(D);
   return UG.ignoreResults();
 }
 
 bool clang::index::generateUSRForMacro(const MacroDefinitionRecord *MD,
                                        const SourceManager &SM,
                                        SmallVectorImpl<char> &Buf) {

clang/lib/Parse/ParseDecl.cpp

 ///
 /// keyword-arg-list:
 ///   keyword-arg
 ///   keyword-arg ',' keyword-arg-list
 ///
 /// keyword-arg:
 ///   'language' '=' <string>
 ///   'defined_in' '=' <string>
+///   'USR' '=' <string>
 ///   'generated_declaration'
 void Parser::ParseExternalSourceSymbolAttribute(
     IdentifierInfo &ExternalSourceSymbol, SourceLocation Loc,
     ParsedAttributes &Attrs, SourceLocation *EndLoc, IdentifierInfo *ScopeName,
     SourceLocation ScopeLoc, ParsedAttr::Syntax Syntax) {
   // Opening '('.
   BalancedDelimiterTracker T(*this, tok::l_paren);
   if (T.expectAndConsume())
     return;
 
   // Initialize the pointers for the keyword identifiers when required.
   if (!Ident_language) {
     Ident_language = PP.getIdentifierInfo("language");
     Ident_defined_in = PP.getIdentifierInfo("defined_in");
     Ident_generated_declaration = PP.getIdentifierInfo("generated_declaration");
+    Ident_USR = PP.getIdentifierInfo("USR");
   }
 
   ExprResult Language;
   bool HasLanguage = false;
   ExprResult DefinedInExpr;
   bool HasDefinedIn = false;
   IdentifierLoc *GeneratedDeclaration = nullptr;
+  ExprResult USR;
+  bool HasUSR = false;
 
   // Parse the language/defined_in/generated_declaration keywords
   do {
     if (Tok.isNot(tok::identifier)) {
       Diag(Tok, diag::err_external_source_symbol_expected_keyword);
       SkipUntil(tok::r_paren, StopAtSemi);
       return;
     }
 
     SourceLocation KeywordLoc = Tok.getLocation();
     IdentifierInfo *Keyword = Tok.getIdentifierInfo();
     if (Keyword == Ident_generated_declaration) {
       if (GeneratedDeclaration) {
         Diag(Tok, diag::err_external_source_symbol_duplicate_clause) << Keyword;
         SkipUntil(tok::r_paren, StopAtSemi);
         return;
       }
       GeneratedDeclaration = ParseIdentifierLoc();
       continue;
     }
 
-    if (Keyword != Ident_language && Keyword != Ident_defined_in) {
+    if (Keyword != Ident_language && Keyword != Ident_defined_in &&
+        Keyword != Ident_USR) {
       Diag(Tok, diag::err_external_source_symbol_expected_keyword);
       SkipUntil(tok::r_paren, StopAtSemi);
       return;
     }
 
     ConsumeToken();
     if (ExpectAndConsume(tok::equal, diag::err_expected_after,
                          Keyword->getName())) {
       SkipUntil(tok::r_paren, StopAtSemi);
       return;
     }
 
-    bool HadLanguage = HasLanguage, HadDefinedIn = HasDefinedIn;
+    bool HadLanguage = HasLanguage, HadDefinedIn = HasDefinedIn,
+         HadUSR = HasUSR;
     if (Keyword == Ident_language)
       HasLanguage = true;
+    else if (Keyword == Ident_USR)
+      HasUSR = true;
     else
       HasDefinedIn = true;
 
     if (Tok.isNot(tok::string_literal)) {
       Diag(Tok, diag::err_expected_string_literal)
           << /*Source='external_source_symbol attribute'*/ 3
-          << /*language | source container*/ (Keyword != Ident_language);
+          << /*language | source container | USR*/ (
+                 Keyword == Ident_language
+                     ? 0
+                     : (Keyword == Ident_defined_in ? 1 : 2));
       SkipUntil(tok::comma, tok::r_paren, StopAtSemi | StopBeforeMatch);
       continue;
     }
     if (Keyword == Ident_language) {
       if (HadLanguage) {
         Diag(KeywordLoc, diag::err_external_source_symbol_duplicate_clause)
             << Keyword;
         ParseStringLiteralExpression();
         continue;
       }
       Language = ParseStringLiteralExpression();
+    } else if (Keyword == Ident_USR) {
+      if (HadUSR) {
+        Diag(KeywordLoc, diag::err_external_source_symbol_duplicate_clause)
+            << Keyword;
+        ParseStringLiteralExpression();
+        continue;
+      }
+      USR = ParseStringLiteralExpression();
     } else {
       assert(Keyword == Ident_defined_in && "Invalid clause keyword!");
       if (HadDefinedIn) {
         Diag(KeywordLoc, diag::err_external_source_symbol_duplicate_clause)
             << Keyword;
         ParseStringLiteralExpression();
         continue;
       }
       DefinedInExpr = ParseStringLiteralExpression();
     }
   } while (TryConsumeToken(tok::comma));
 
   // Closing ')'.
   if (T.consumeClose())
     return;
   if (EndLoc)
     *EndLoc = T.getCloseLocation();
 
-  ArgsUnion Args[] = {Language.get(), DefinedInExpr.get(),
-                      GeneratedDeclaration};
+  ArgsUnion Args[] = {Language.get(), DefinedInExpr.get(), GeneratedDeclaration,
+                      USR.get()};
   Attrs.addNew(&ExternalSourceSymbol, SourceRange(Loc, T.getCloseLocation()),
                ScopeName, ScopeLoc, Args, std::size(Args), Syntax);
 }
 
 /// Parse the contents of the "objc_bridge_related" attribute.
 /// objc_bridge_related '(' related_class ',' opt-class_method ',' opt-instance_method ')'
 /// related_class:
 ///     Identifier

clang/lib/Parse/Parser.cpp

void Parser::Initialize() {
 
   Ident_introduced = nullptr;
   Ident_deprecated = nullptr;
   Ident_obsoleted = nullptr;
   Ident_unavailable = nullptr;
   Ident_strict = nullptr;
   Ident_replacement = nullptr;
 
-  Ident_language = Ident_defined_in = Ident_generated_declaration = nullptr;
+  Ident_language = Ident_defined_in = Ident_generated_declaration = Ident_USR =
+      nullptr;
 
   Ident__except = nullptr;
 
   Ident__exception_code = Ident__exception_info = nullptr;
   Ident__abnormal_termination = Ident___exception_code = nullptr;
   Ident___exception_info = Ident___abnormal_termination = nullptr;
   Ident_GetExceptionCode = Ident_GetExceptionInfo = nullptr;
   Ident_AbnormalTermination = nullptr;

clang/lib/Sema/SemaDeclAttr.cpp

if (NewII) {
         }
       }
     }
   }
 }
 
 static void handleExternalSourceSymbolAttr(Sema &S, Decl *D,
                                            const ParsedAttr &AL) {
-  if (!AL.checkAtLeastNumArgs(S, 1) || !AL.checkAtMostNumArgs(S, 3))
+  if (!AL.checkAtLeastNumArgs(S, 1) || !AL.checkAtMostNumArgs(S, 4))
     return;
 
   StringRef Language;
   if (const auto *SE = dyn_cast_or_null<StringLiteral>(AL.getArgAsExpr(0)))
     Language = SE->getString();
   StringRef DefinedIn;
   if (const auto *SE = dyn_cast_or_null<StringLiteral>(AL.getArgAsExpr(1)))
     DefinedIn = SE->getString();
   bool IsGeneratedDeclaration = AL.getArgAsIdent(2) != nullptr;
+  StringRef USR;
+  if (const auto *SE = dyn_cast_or_null<StringLiteral>(AL.getArgAsExpr(3)))
+    USR = SE->getString();
 
   D->addAttr(::new (S.Context) ExternalSourceSymbolAttr(
-      S.Context, AL, Language, DefinedIn, IsGeneratedDeclaration));
+      S.Context, AL, Language, DefinedIn, IsGeneratedDeclaration, USR));
 }
 
 template <class T>
 static T *mergeVisibilityAttr(Sema &S, Decl *D, const AttributeCommonInfo &CI,
                               typename T::VisibilityType value) {
   T *existingAttr = D->getAttr<T>();
   if (existingAttr) {
     typename T::VisibilityType existingValue = existingAttr->getVisibility();

clang/test/AST/ast-dump-attr.cpp

 void TestExternalSourceSymbolAttr1()
 __attribute__((external_source_symbol(language="Swift", defined_in="module", generated_declaration)));
 // CHECK: FunctionDecl{{.*}} TestExternalSourceSymbolAttr1
 // CHECK-NEXT: ExternalSourceSymbolAttr{{.*}} "Swift" "module" GeneratedDeclaration
 
 void TestExternalSourceSymbolAttr2()
 __attribute__((external_source_symbol(defined_in="module", language="Swift")));
 // CHECK: FunctionDecl{{.*}} TestExternalSourceSymbolAttr2
-// CHECK-NEXT: ExternalSourceSymbolAttr{{.*}} "Swift" "module"{{$}}
+// CHECK-NEXT: ExternalSourceSymbolAttr{{.*}} "Swift" "module" ""{{$}}
 
 void TestExternalSourceSymbolAttr3()
 __attribute__((external_source_symbol(generated_declaration, language="Objective-C++", defined_in="module")));
 // CHECK: FunctionDecl{{.*}} TestExternalSourceSymbolAttr3
 // CHECK-NEXT: ExternalSourceSymbolAttr{{.*}} "Objective-C++" "module" GeneratedDeclaration
 
 void TestExternalSourceSymbolAttr4()
 __attribute__((external_source_symbol(defined_in="Some external file.cs", generated_declaration, language="C Sharp")));
 // CHECK: FunctionDecl{{.*}} TestExternalSourceSymbolAttr4
 // CHECK-NEXT: ExternalSourceSymbolAttr{{.*}} "C Sharp" "Some external file.cs" GeneratedDeclaration
 
 void TestExternalSourceSymbolAttr5()
 __attribute__((external_source_symbol(generated_declaration, defined_in="module", language="Swift")));
 // CHECK: FunctionDecl{{.*}} TestExternalSourceSymbolAttr5
 // CHECK-NEXT: ExternalSourceSymbolAttr{{.*}} "Swift" "module" GeneratedDeclaration
 
+void TestExternalSourceSymbolAttr6()
+__attribute__((external_source_symbol(generated_declaration, defined_in="module", language="Swift", USR="testUSR")));
+// CHECK: FunctionDecl{{.*}} TestExternalSourceSymbolAttr6
+// CHECK-NEXT: ExternalSourceSymbolAttr{{.*}} "Swift" "module" GeneratedDeclaration "testUSR"
+
 namespace TestNoEscape {
   void noescapeFunc(int *p0, __attribute__((noescape)) int *p1) {}
   // CHECK: `-FunctionDecl{{.*}} noescapeFunc 'void (int *, __attribute__((noescape)) int *)'
   // CHECK-NEXT: ParmVarDecl
   // CHECK-NEXT: ParmVarDecl
   // CHECK-NEXT: NoEscapeAttr
 }
 

clang/test/Index/Core/external-source-symbol-attr-cxx.cpp

+// RUN: c-index-test core -print-source-symbols -- %s -target x86_64-apple-macosx10.7 | FileCheck %s
+
+#define GEN_DECL_USR(mod_name, usr) __attribute__((external_source_symbol(language="Swift", defined_in=mod_name, USR=usr, generated_declaration)))
+
+class GEN_DECL_USR("Module", "s:Class") Class {
+public:
+    void method() GEN_DECL_USR("Module", "s:Class_method");
+    void method2() GEN_DECL_USR("Module", "");
+
+    static void staticMethod() GEN_DECL_USR("Module", "s:Class_staticMethod");
+};
+
+template<class T>
+class GEN_DECL_USR("Module", "s:TemplateClass") TemplateClass {
+public:
+    void method() GEN_DECL_USR("Module", "s:TemplateClass_method");
+};
+
+void test() {
+  Class c = Class();
+  // CHECK: [[@LINE-1]]:3 | class/Swift | Class | s:Class |
+  // CHECK: [[@LINE-2]]:13 | class/Swift | Class | s:Class |
+  c.method();
+  // CHECK: [[@LINE-1]]:5 | instance-method/Swift | method | s:Class_method |
+  c.method2();
+  // CHECK: [[@LINE-1]]:5 | instance-method/Swift | method2 | c:@M@Module@S@Class@F@method2# |
+  Class::staticMethod();
+  // CHECK: [[@LINE-1]]:10 | static-method/Swift | staticMethod | s:Class_staticMethod |
+  // CHECK: [[@LINE-2]]:3 | class/Swift | Class | s:Class |
+  TemplateClass<int> c2 = TemplateClass<int>();
+  // CHECK: [[@LINE-1]]:3 | class(Gen)/Swift | TemplateClass | s:TemplateClass |
+  // CHECK: [[@LINE-2]]:27 | class(Gen)/Swift | TemplateClass | s:TemplateClass |
+  c2.method();
+  // CHECK: [[@LINE-1]]:6 | instance-method/Swift | method | s:TemplateClass_method |
+}

clang/test/Index/Core/external-source-symbol-attr.m

 // RUN: c-index-test core -print-source-symbols -- %s -target x86_64-apple-macosx10.7 | FileCheck %s
 
 #define EXT_DECL(mod_name) __attribute__((external_source_symbol(language="Swift", defined_in=mod_name)))
 #define GEN_DECL(mod_name) __attribute__((external_source_symbol(language="Swift", defined_in=mod_name, generated_declaration)))
 #define PUSH_GEN_DECL(mod_name) push(GEN_DECL(mod_name), apply_to=any(enum, objc_interface, objc_category, objc_protocol))
 
+#define GEN_DECL_USR(mod_name, usr) __attribute__((external_source_symbol(language="Swift", defined_in=mod_name, USR=usr, generated_declaration)))
+
 // Forward declarations should not affect module namespacing below
 @class I1;
 @class I2;
 
 // This should not be indexed.
 GEN_DECL("some_module")
 @interface I1
 // CHECK-NOT: [[@LINE-1]]:12 |

 -(void)meth_other_mod;
 @end
 #pragma clang attribute pop
 
 void test3(I3 *i3) {
   [i3 meth_other_mod];
   // CHECK: [[@LINE-1]]:7 | instance-method/Swift | meth_other_mod | c:@CM@other_mod_for_cat@modname@objc(cs)I3(im)meth_other_mod |
 }
+
+void function() GEN_DECL_USR("SwiftMod", "s:8SwiftMod8functionyyF");
+
+void test4() {
+  function();
+  // CHECK: [[@LINE-1]]:3 | function/Swift | function | s:8SwiftMod8functionyyF
+}

clang/test/Parser/attr-external-source-symbol.m

 // RUN: %clang_cc1 -fsyntax-only -verify %s
 
 void function(void) __attribute__((external_source_symbol(language="Swift", defined_in="module", generated_declaration)));
 
 __attribute__((external_source_symbol(language="Swift", defined_in="module")))
 @interface I
 
 - (void)method __attribute__((external_source_symbol(defined_in= "module")));
 
 @end
 
 enum E {
   CaseA __attribute__((external_source_symbol(generated_declaration))),
   CaseB __attribute__((external_source_symbol(generated_declaration, language="Swift")))
 } __attribute__((external_source_symbol(language = "Swift")));
 
+void functionCustomUSR(void) __attribute__((external_source_symbol(language="Swift", defined_in="module", generated_declaration, USR="s:6module17functionCustomUSRyyF")));
+
+void functionCustomUSR2(void) __attribute__((external_source_symbol(language="Swift", defined_in="module", USR="s:6module18functionCustomUSR2yyF", generated_declaration)));
+
 void f2(void)
-__attribute__((external_source_symbol())); // expected-error {{expected 'language', 'defined_in', or 'generated_declaration'}}
+__attribute__((external_source_symbol())); // expected-error {{expected 'language', 'defined_in', 'generated_declaration', or 'USR'}}
 void f3(void)
-__attribute__((external_source_symbol(invalid))); // expected-error {{expected 'language', 'defined_in', or 'generated_declaration'}}
+__attribute__((external_source_symbol(invalid))); // expected-error {{expected 'language', 'defined_in', 'generated_declaration', or 'USR'}}
 void f4(void)
 __attribute__((external_source_symbol(language))); // expected-error {{expected '=' after language}}
 void f5(void)
 __attribute__((external_source_symbol(language=))); // expected-error {{expected string literal for language name in 'external_source_symbol' attribute}}
 void f6(void)
 __attribute__((external_source_symbol(defined_in=20))); // expected-error {{expected string literal for source container name in 'external_source_symbol' attribute}}
 
 void f7(void)
 __attribute__((external_source_symbol(generated_declaration, generated_declaration))); // expected-error {{duplicate 'generated_declaration' clause in an 'external_source_symbol' attribute}}
 void f8(void)
 __attribute__((external_source_symbol(language="Swift", language="Swift"))); // expected-error {{duplicate 'language' clause in an 'external_source_symbol' attribute}}
 void f9(void)
 __attribute__((external_source_symbol(defined_in="module", language="Swift", defined_in="foo"))); // expected-error {{duplicate 'defined_in' clause in an 'external_source_symbol' attribute}}
+__attribute__((external_source_symbol(defined_in="module", language="Swift", USR="foo", USR="bar"))); // expected-error {{duplicate 'USR' clause in an 'external_source_symbol' attribute}}
+void f9_1(void);
 
 void f10(void)
 __attribute__((external_source_symbol(generated_declaration, language="Swift", defined_in="foo", generated_declaration, generated_declaration, language="Swift"))); // expected-error {{duplicate 'generated_declaration' clause in an 'external_source_symbol' attribute}}
 
 void f11(void)
 __attribute__((external_source_symbol(language="Objective-C++", defined_in="Some file with spaces")));
 
 void f12(void)
 __attribute__((external_source_symbol(language="C Sharp", defined_in="file:////Hello world with spaces. cs")));
 
 void f13(void)
 __attribute__((external_source_symbol(language=Swift))); // expected-error {{expected string literal for language name in 'external_source_symbol' attribute}}
 
 void f14(void)
-__attribute__((external_source_symbol(=))); // expected-error {{expected 'language', 'defined_in', or 'generated_declaration'}}
+__attribute__((external_source_symbol(=))); // expected-error {{expected 'language', 'defined_in', 'generated_declaration', or 'USR'}}
 
 void f15(void)
-__attribute__((external_source_symbol(="Swift"))); // expected-error {{expected 'language', 'defined_in', or 'generated_declaration'}}
+__attribute__((external_source_symbol(="Swift"))); // expected-error {{expected 'language', 'defined_in', 'generated_declaration', or 'USR'}}
 
 void f16(void)
-__attribute__((external_source_symbol("Swift", "module", generated_declaration))); // expected-error {{expected 'language', 'defined_in', or 'generated_declaration'}}
+__attribute__((external_source_symbol("Swift", "module", generated_declaration))); // expected-error {{expected 'language', 'defined_in', 'generated_declaration', or 'USR'}}
 
 void f17(void)
-__attribute__((external_source_symbol(language="Swift", "generated_declaration"))); // expected-error {{expected 'language', 'defined_in', or 'generated_declaration'}}
+__attribute__((external_source_symbol(language="Swift", "generated_declaration"))); // expected-error {{expected 'language', 'defined_in', 'generated_declaration', or 'USR'}}
 
 void f18(void)
 __attribute__((external_source_symbol(language= =))); // expected-error {{expected string literal for language name in 'external_source_symbol' attribute}}
 
 void f19(void)
 __attribute__((external_source_symbol(defined_in="module" language="swift"))); // expected-error {{expected ')'}} expected-note {{to match this '('}}
 
 void f20(void)

 
 void f24(void)
 __attribute__((external_source_symbol(language = generated_declaration))); // expected-error {{expected string literal for language name in 'external_source_symbol' attribute}}
 
 void f25(void)
 __attribute__((external_source_symbol(defined_in=123, defined_in="module"))); // expected-error {{expected string literal for source container name in 'external_source_symbol'}} expected-error {{duplicate 'defined_in' clause in an 'external_source_symbol' attribute}}
 
 void f26(void)
-__attribute__((external_source_symbol(language=Swift, language="Swift", error))); // expected-error {{expected string literal for language name in 'external_source_symbol'}} expected-error {{duplicate 'language' clause in an 'external_source_symbol' attribute}} expected-error {{expected 'language', 'defined_in', or 'generated_declaration'}}
+__attribute__((external_source_symbol(language=Swift, language="Swift", error))); // expected-error {{expected string literal for language name in 'external_source_symbol'}} expected-error {{duplicate 'language' clause in an 'external_source_symbol' attribute}} expected-error {{expected 'language', 'defined_in', 'generated_declaration', or 'USR'}}
+
+void f27(void)
+__attribute__((external_source_symbol(USR=f27))); // expected-error {{expected string literal for USR in 'external_source_symbol' attribute}}
+
+void f28(void)
+__attribute__((external_source_symbol(USR="")));
+
+#if __has_attribute(external_source_symbol) != 20230206
+# error "invalid __has_attribute version"
+#endif

clang/test/Sema/attr-external-source-symbol-cxx.cpp

+// RUN: %clang_cc1 -fsyntax-only -fblocks -verify -fdouble-square-bracket-attributes %s
+
+template<class T>
+class Class {
+public:
+    [[clang::external_source_symbol(language="Swift", defined_in="module", USR="test", generated_declaration)]]
+    void testExternalSourceSymbol();
+
+    // expected-error@+1 {{expected string literal for USR in 'external_source_symbol' attribute}}
+    [[clang::external_source_symbol(language="Swift", defined_in="module", USR=T, generated_declaration)]]
+    void testExternalSourceSymbol2();
+};
+
+template<class T>
+void Class<T>::testExternalSourceSymbol() {
+}

clang/test/Sema/attr-external-source-symbol.c

 // RUN: %clang_cc1 -fsyntax-only -fblocks -verify -fdouble-square-bracket-attributes %s
 
 void threeClauses(void) __attribute__((external_source_symbol(language="Swift", defined_in="module", generated_declaration)));
 
 void twoClauses(void) __attribute__((external_source_symbol(language="Swift", defined_in="module")));
 
-void fourClauses(void) __attribute__((external_source_symbol(language="Swift", defined_in="module", generated_declaration, generated_declaration))); // expected-error {{duplicate 'generated_declaration' clause in an 'external_source_symbol' attribute}}
+void fourClauses(void) __attribute__((external_source_symbol(language="Swift", defined_in="module", generated_declaration, USR="test")));
+
+void fiveClauses(void) __attribute__((external_source_symbol(language="Swift", defined_in="module", generated_declaration, generated_declaration, USR="test"))); // expected-error {{duplicate 'generated_declaration' clause in an 'external_source_symbol' attribute}}
 
 void oneClause(void) __attribute__((external_source_symbol(generated_declaration)));
 
 void noArguments(void)
 __attribute__((external_source_symbol)); // expected-error {{'external_source_symbol' attribute takes at least 1 argument}}
 
 void namedDeclsOnly(void) {
   int (^block)(void) = ^ (void)
     __attribute__((external_source_symbol(language="Swift"))) { // expected-warning {{'external_source_symbol' attribute only applies to named declarations}}
       return 1;
   };
 }
 
 [[clang::external_source_symbol(language="Swift", defined_in="module", generated_declaration)]] void threeClauses2(void);
 
 [[clang::external_source_symbol(language="Swift", defined_in="module")]] void twoClauses2(void);
 
-[[clang::external_source_symbol(language="Swift", defined_in="module", generated_declaration, generated_declaration)]] // expected-error {{duplicate 'generated_declaration' clause in an 'external_source_symbol' attribute}}
-void fourClauses2(void);
+[[clang::external_source_symbol(language="Swift", defined_in="module", USR="test", generated_declaration, generated_declaration)]] // expected-error {{duplicate 'generated_declaration' clause in an 'external_source_symbol' attribute}}
+void fiveClauses2(void);
 
 [[clang::external_source_symbol(generated_declaration)]] void oneClause2(void);
 
 [[clang::external_source_symbol]] // expected-error {{'external_source_symbol' attribute takes at least 1 argument}}
 void noArguments2(void);

clang/utils/TableGen/ClangAttrEmitter.cpp

 
 using namespace llvm;
 
 namespace {
 
 class FlattenedSpelling {
   std::string V, N, NS;
   bool K = false;
+  const Record &OriginalSpelling;
 
 public:
   FlattenedSpelling(const std::string &Variety, const std::string &Name,
-                    const std::string &Namespace, bool KnownToGCC) :
-    V(Variety), N(Name), NS(Namespace), K(KnownToGCC) {}
+                    const std::string &Namespace, bool KnownToGCC,
+                    const Record &OriginalSpelling)
+      : V(Variety), N(Name), NS(Namespace), K(KnownToGCC),
+        OriginalSpelling(OriginalSpelling) {}
   explicit FlattenedSpelling(const Record &Spelling)
       : V(std::string(Spelling.getValueAsString("Variety"))),
-        N(std::string(Spelling.getValueAsString("Name"))) {
+        N(std::string(Spelling.getValueAsString("Name"))),
+        OriginalSpelling(Spelling) {
     assert(V != "GCC" && V != "Clang" &&
            "Given a GCC spelling, which means this hasn't been flattened!");
     if (V == "CXX11" || V == "C2x" || V == "Pragma")
       NS = std::string(Spelling.getValueAsString("Namespace"));
   }
 
   const std::string &variety() const { return V; }
   const std::string &name() const { return N; }
   const std::string &nameSpace() const { return NS; }
   bool knownToGCC() const { return K; }
+  const Record &getSpellingRecord() const { return OriginalSpelling; }
 };
 
 } // end anonymous namespace
 
 static std::vector<FlattenedSpelling>
 GetFlattenedSpellings(const Record &Attr) {
   std::vector<Record *> Spellings = Attr.getValueAsListOfDefs("Spellings");
   std::vector<FlattenedSpelling> Ret;
 
   for (const auto &Spelling : Spellings) {
     StringRef Variety = Spelling->getValueAsString("Variety");
     StringRef Name = Spelling->getValueAsString("Name");
     if (Variety == "GCC") {
-      Ret.emplace_back("GNU", std::string(Name), "", true);
-      Ret.emplace_back("CXX11", std::string(Name), "gnu", true);
+      Ret.emplace_back("GNU", std::string(Name), "", true, *Spelling);
+      Ret.emplace_back("CXX11", std::string(Name), "gnu", true, *Spelling);
       if (Spelling->getValueAsBit("AllowInC"))
-        Ret.emplace_back("C2x", std::string(Name), "gnu", true);
+        Ret.emplace_back("C2x", std::string(Name), "gnu", true, *Spelling);
     } else if (Variety == "Clang") {
-      Ret.emplace_back("GNU", std::string(Name), "", false);
-      Ret.emplace_back("CXX11", std::string(Name), "clang", false);
+      Ret.emplace_back("GNU", std::string(Name), "", false, *Spelling);
+      Ret.emplace_back("CXX11", std::string(Name), "clang", false, *Spelling);
       if (Spelling->getValueAsBit("AllowInC"))
-        Ret.emplace_back("C2x", std::string(Name), "clang", false);
+        Ret.emplace_back("C2x", std::string(Name), "clang", false, *Spelling);
     } else
       Ret.push_back(FlattenedSpelling(*Spelling));
   }
 
   return Ret;
 }
 
 static std::string ReadPCHRecord(StringRef type) {
for (const auto *Attr : Attrs) {
     // have the default value (1), as that's incorrect. Instead, the unscoped
     // attribute version information should be taken from the SD-6 standing
     // document, which can be found at:
     // https://isocpp.org/std/standing-documents/sd-6-sg10-feature-test-recommendations
     //
     // C2x-style attributes have the same kind of version information
     // associated with them. The unscoped attribute version information should
     // be taken from the specification of the attribute in the C Standard.
+    //
+    // Clang-specific attributes have the same kind of version information
+    // associated with them. This version is typically the default value (1).
+    // These version values are clang-specific and should typically be
+    // incremented once the attribute changes its syntax and/or semantics in a
+    // a way that is impactful to the end user.
     int Version = 1;
 
-    if (Variety == "CXX11" || Variety == "C2x") {
-      std::vector<Record *> Spellings = Attr->getValueAsListOfDefs("Spellings");
-      for (const auto &Spelling : Spellings) {
-        if (Spelling->getValueAsString("Variety") == Variety) {
-          Version = static_cast<int>(Spelling->getValueAsInt("Version"));
-          if (Scope.empty() && Version == 1)
-            PrintError(Spelling->getLoc(), "Standard attributes must have "
-                                           "valid version information.");
-          break;
-        }
-      }
-    }
+    std::vector<FlattenedSpelling> Spellings = GetFlattenedSpellings(*Attr);
+      if (Spelling.variety() == Variety &&
+          (Spelling.nameSpace().empty() || Scope == Spelling.nameSpace())) {
+        Version = static_cast<int>(
+            Spelling.getSpellingRecord().getValueAsInt("Version"));
+        // Verify that explicitly specified CXX11 and C2x spellings (i.e.
+        // not inferred from Clang/GCC spellings) have a version that's
+        // different than the default (1).
+        bool RequiresValidVersion =
+            (Variety == "CXX11" || Variety == "C2x") &&
+            Spelling.getSpellingRecord().getValueAsString("Variety") == Variety;
+        if (RequiresValidVersion && Scope.empty() && Version == 1)
+          PrintError(Spelling.getSpellingRecord().getLoc(),
+                     "Standard attributes must have "
 
     std::string Test;
     if (Attr->isSubClassOf("TargetSpecificAttr")) {
       const Record *R = Attr->getValueAsDef("Target");
       std::vector<StringRef> Arches = R->getValueAsListOfStrings("Arches");
       GenerateTargetSpecificAttrChecks(R, Arches, Test, nullptr);
 
       // If this is the C++11 variety, also add in the LangOpts test.
       if (Variety == "CXX11")
         Test += " && LangOpts.CPlusPlus11";
       else if (Variety == "C2x")
         Test += " && LangOpts.DoubleSquareBracketAttributes";
     } else if (Variety == "CXX11")
       // C++11 mode should be checked against LangOpts, which is presumed to be
       // present in the caller.
       Test = "LangOpts.CPlusPlus11";
     else if (Variety == "C2x")
       Test = "LangOpts.DoubleSquareBracketAttributes";
 
-    std::string TestStr =
-        !Test.empty() ? Test + " ? " + llvm::itostr(Version) + " : 0" : "1";
-    std::vector<FlattenedSpelling> Spellings = GetFlattenedSpellings(*Attr);
+    std::string TestStr = !Test.empty()
+                              ? Test + " ? " + llvm::itostr(Version) + " : 0"
+                              : llvm::itostr(Version);
     for (const auto &S : Spellings)
       if (Variety.empty() || (Variety == S.variety() &&
                               (Scope.empty() || Scope == S.nameSpace())))
         OS << "    .Case(\"" << S.name() << "\", " << TestStr << ")\n";
   }
   OS << "    .Default(0);\n";
 }